Replicated Queries

MS-Access makes it very easy to change the SQL of a query simply by using a statement like

CurrentDb.QueryDefs("MyQuery").SQL = "SELECT * FROM MyTable"

there is no need to Dim or Set anything, simply plug in the SQL and you're ready to go.

However, I just discovered that this does not work in a replicated database. Or I should say, it works in the Design Master but it does not work in a Replica. I needed to apply some complex criteria, more complex than could be expressed simply by using a WHERE condition when opening a report. Testing the code in the DM worked just fine, but users received a message "cannot update...object is read-only" because Access considers this to be a design change & does not allow that in a Replica.

You can code things like changing visibility, font colors, enabling & disabling controls, but QueryDefs are off limits.

The challenge: I had an existing report that used a query for the detail section, and the report footer contained a subreport with a summary that used the same query as the body of the report. Opening the report with a filter or where argument would properly filter the detail section, but the subreport would display all of the data. I tried using the OnOpen or OnFormat event to apply a filter to the subreport but that did not work.

(EDIT) At this point, I tried to change the SQL for the QueryDef using the type of statement shown above, to filter the data for both the body of the report and the summary subreport; this worked in the DM but not the Replica. My original post has not been clear about that. (EDIT)

My solution was basically to create a staging table for the subreport, and flush-and-fill the table prior to opening the report.

In the DM I did this:

• Ran the old (bad) code to set up the SQL statement
• Switched the query to a make-table, to create the staging table
• Switched the query to an append query
• Copied the append SQL and used that to build up a dynamic SQL statement in VBA

The button OnClick would then delete all records from the staging table, run the dynamic SQL to populate that table, and then open the report.

Moving Days

{EDIT} Around about September 2008, I decided not to move articles off this blog after all. I might copy a few to the tips page at my Web site, but otherwise I'm going to keep this blog intact. {EDIT}

When I started this blog three years ago, I had intended to use it as a quick way to publish my technology findings both for the general Internet community, and also to use it as a knowledge base for myself.

As it turns out, the older articles wind up being buried in the archives, grouped by month, so many times I've had to guess the month & drill down from there. The built-in search is quite effective, but being able to scan down a complete list of topics is much faster.

For those reasons, I've been moving articles from blogger to my personal web site, where I use a classic ASP page and the FileSystemObject to present a scrollable list of all articles, sizes, dates etc. The current list can be found at http://www.wvmitchell.com/tips.

Another difficulty using the blogger format is that many html tags are scrubbed-out from posts, so any code samples will lose their indenting & formatting, making them hard to decipher. My workaround has been to present a link to the actual code on a separate web page, but I would prefer to be able to post a complete article intact with all the formatting.

The blogger search facilities work very well, and Google indexes the blogger posts, so much information can be found via a normal Google search, but my web site also includes a search feature so nothing will be lost.

In closing, this blog will eventually be shrunk down to include only tech comments and observations.

Batch Files

By all accounts, DOS is dead. But since cmd.exe is still part of Windows, you can still write DOS batch programs to automate certain tasks.

For example, I am working with an Access application that is rebuilt every month. The previous process was to have the user navigate to a shared drive and use Windows Explorer to copy a new version of the MDB file and then paste it to a specific folder on their C: drive.

The new process uses a simple batch file like this:

@ECHO OFF
CLS
ECHO.
ECHO *** XYZ file setup program ***
ECHO.
IF NOT EXIST C:\XYZ MD C:\XYZ

IF NOT EXIST C:\XYZ\XYZ.MDB GOTO NOFILE
ECHO.
ECHO You already have a copy of XYZ on your computer
ECHO ===============================================
ECHO.
ECHO  Do you want to replace your existing copy?
ECHO  ------------------------------------------
GOTO HASFILE

:NOFILE
ECHO *** COPYING FILE, PLEASE WAIT ***
COPY /Y \\MYSERVER\XYZ\XYZ.MDB C:\XYZ
GOTO END

:HASFILE
ECHO  Y = Yes, replace my old file
ECHO  N = No, keep my existing copy of the file
ECHO.
SET CHOICE=
SET /P CHOICE= Type the letter and press {ENTER}
IF /I '%CHOICE%'=='Y' GOTO NOFILE
ECHO.
ECHO  (Nothing was copied or replaced)
ECHO.
:END
PAUSE
EXIT

For a new user, the folder is created if it does not exist.
Then, if the file exists the user is prompted to either overwrite or keep their existing copy.

Without the CHOICE statement, DOS would ask the overwrite question as
Yes/No/All
and this avoids the confusing "All" option.

Removing Access Replication

Using replication with MS-Access adds numerous system fields and tables, and combined with the replication-tracking system, a replicated Access database can grow to an unwieldy size, especially if you have many users and/or there are frequent changes in the design master.

The customary solution is to rebuild the database, i.e. to create a new non-replicated file and then use that to create a new design master and the replicas.

Unfortunately, you cannot simply create a new empty file and import all your Access objects, because all the system fields in the tables will also be carried over via the import. You cannot simply go into design view & delete those replication fields because they are system fields.

The basic process is to create a new empty file, and then export all the Access objects except the tables, and then export the tables but without the replication fields.

Import the VBA module Unreplicate_Access into your design master, and when you execute the code it will create a new non-replicated version of your database. It will also set a number of startup properties such as hiding the DB Window, turning off shortcut keys and so on.

Once you have created the new file, simply copy it over your existing design master (after you make a backup copy, of course) and then click Tools ~ Replication ~ Create a Replica and follow the prompts.

This module is still a work in progress; it does not set any primary keys, nor does it copy any default values for the tables. This will appear in a future post. But there won't be any code to create relationships; Microsoft actually recommends against using RI in a replicated db due to the mechanics of replication.

SSMA, part 3

My previous posts about the SQL Migration Assistant for Access reported generally favorable results with this product which is used to migrate Access data into SQL Server. It does a remarkably better job than the old Access Upsizing Wizard which very often failed, especially when dealing with date fields. When SSMA has completed its work, the system is mostly operational.

However, there are two common Access design flaws that can impact the application - the first is relying on the behavior of Boolean fields to evaluate as -1 = True and 0 = False. Many Access developers will test for "-1" meaning True, but SQL Server returns "1" for bit fields so the comparison logic fails. The fix is simply to substitute "-1" with "True" (without the quotes) and the operation will succeed.

A second design flaw recently noted is that in Access you can subtract one date from another date, and then perform math on the result - for example

24 * ([date_end] - [date_start])

in Access will give you the number of hours between the two date/time values. However, once your data is moved into SQL Server, this expression will result in

Implicit conversion from data type smalldatetime to int is not allowed. Use the CONVERT function to run this query.

The solution is to use the DateDiff function which exists in both Access and SQL Server. In Access you would write

DateDiff("h", [date_start], [date_end])

and when converting this query to T-SQL for SQL Server you might use

DATEDIFF(hh, [date_start], [date_end])

Access Replication

Replication in Access provides a method of allowing remote users to work with a local copy of the database while disconnected from the network, and then synchronizing their changes when they do connect. This feature has been part of Access since the 90's but sadly, although Access 2007 supports it if you stay with the .mdb file format, if you choose to convert to the new .accdb format this feature goes away (along with WorkGroup Security).

The idea of replication is that you create a Design Master (DM) that adds additional hidden system fields which Access uses to track record changes that need to be replicated. You also create a Replica which has all the programming and data of the DM but without any design options, thus protecting the design from unauthorized changes. During the sync process, data is exchanged between the DM and replica, and design changes made to the DM are also propagated to the replica.

The replication model is sensitive to the drive, path, and file name; moving the DM will cause it to become a replica; copying the DM will result in creating another replica.

In addition to the fields added to each table, there are also a number of hidden system tables created when you replicate a database; the net result is that the DM and the replicas are considerably larger than the original non-replicated file. For example, a 27 MB file can grow to 44 MB when replicated. Once the users begin working with the file and syncing, that file can grow to 48-59 MB (these numbers are taken from an actual production system). For that reason, it is strongly recommended that you enable the Compact on Close option to keep the file as small as possible under the circumstances.

It is important to note that creating a replica will change AutoNumber fields from Increment to Random. This is necessary to prevent conflicts - for example, if user A added a new record, and user B added a new record, Increment would otherwise result in a key violation when attempting to sync. Therefore, if the app relies on a sequential series of new numbers, e.g. invoice numbers, you will need to develop a different strategy for that numbering system.

During the sync process, any conflicts will result in a hidden table being created, e.g. if a table tblOrders has sync conflicts then Access will create a tblOrders_Conflict table that holds the GUIDS for the "winning" and "losing" replica, table, and field.

It should be noted that simply deleting old records will not necessarily reduce the file size; Access maintains a system table, MSysTombstone, which retains a pointer to every deleted record. Unless you utilize Replication Manager (from the Office 2000 developer kit) or a third-party utility, the default retention period is 1000 days, so all those deleted records will be adding to the size of the file with little real benefit.

A common technique of dealing with the bloat of replication is to rebuild the master file periodically. To accomplish this, you first have all users sync & then delete their local copy, then you would need to create a new empty file, export all the Access objects into the new file, and also create new tables in the new file that do not have the replication fields & export the data into those new tables. (It is not possible to delete the replication fields from the tables because they are system fields.) The final steps would be to convert that new file to a design master & create a replica, and then distribute that replica to the database users.

SSMA, part 2

The SQL Server Migration Assistant for Access does a fine job in converting Access data into a SQL Server database. But there are a few issues.

One issue is that SSMA does not convert relationships where the PrimaryKey in Access is a compound index, i.e. if your PK uses more than one field, that relationship will not migrate so you'll have to create that manually.

Another issue is that SSMA does not convert relationships that do not enforce referential integrity (RI). Not a common scenario, but if you do have those relationships in Access then you'll need to create them manually in SQL Server.

A third issue relates to the Access application itself. Access represents boolean fields as -1 = True, and 0 = False. It is very common for Access developers to use this property in code or queries, for example, testing if SomeField=-1, rather than testing SomeField=True.

This will not work correctly after using SSMA, because SQL Server represents 1 = True. To fix this, you will need to inspect every query to find those occurrences and change SomeField=True. This should be done in Access before using SSMA, to preserve the Access code base so that the Access-data version is the same as the MSSQL-data version.

One final issue, if you want to call it that, is that SSMA retains all of the Access tables but renames them by bracketing the name with SSMA$ and $local. If you have a large amount of data, the resulting Access file will be somewhat larger after conversion because the linked tables add a small amount of bulk to the file. After the first conversion, it is useful to have the Access tables in the event that something does not convert as expected, but if you are using an iterative process - convert the data, test the app, modify the app as required, repeat the conversion - there is no option I found in SSMA to delete the original Access tables. That would be a nice addition to the product.

SQL Server Migration Assistant

"Microsoft SQL Server Migration Assistant (SSMA) for Access is a tool for migrating databases from Microsoft Access 97 through Access 2003 to Microsoft SQL Server 2005. SSMA for Access converts Access database objects to SQL Server database objects, loads those objects into SQL Server, and then migrates data from Access to SQL Server." (from the readme file)

This is an extremely useful tool, the SSMA does a very nice job of quickly upsizing an Access 97 - 2003 database to SQL Server, creating the tables and relationships & migrating the data, with just a few clicks.

It renames the existing Access files, for example tblCities becomes SSMA$tblCities$local so your original data is preserved until you're satisfied that the upsize worked as expected, and the linked tables assume the names of the original tables e.g. tblCities, omitting the "dbo_" prefix that you normally see when linking to MSSQL tables.

Note that you must have a primary key defined for every table, otherwise it only grants SELECT permissions on those un-keyed tables.

The only possible downside is the way that auto-numbering works - in Access, an Autonumber value exists as soon as you begin to create a new record, but in SQL Server an IDENTITY value does not exist until the new record is saved; if your Access app depends on that behavior, then you'll need some re-write in your coding.

Oh yes, you need to install the J# 2.0 redistributable package prior to installing SSMA but the installer alerts you and provides a quick link to download & install it.

There is also a CTP available for migrating the new Access 2007 to MSSQL 2005, and another CTP for migrating to MSSQL 2008 that also supports Access 2007.

Office 2007 creates PDF's

There is a significant new feature in Office 2007 that may encourage users to migrate to the 2007 version - the ability to create PDF files without requiring the full version of Adobe Acrobat. This is especially useful with Access 2007.

In previous Office versions, you would need to install a PDF-creation program such as Acrobat, pdf995, or other program. Those programs generally install themselves as a printer, so for an Access report you would need to create a copy of the desired report, and set the default printer to be the PDF creator. Even then, the PDF program would prompt the user to navigate to the desired drive & folder, and supply a name for the PDF to be generated.

It should be noted that with previous versions, it is possible to use the same report, but run some complex code to modify the PrintDev settings for the report to "print" to the PDF creator; this code can be found in Ken Getz's Access Developer's Handbook, among other sources. But this is no longer necessary with Office 2007.

Microsoft has published a free add-in to create either PDF or XPS format files for Office 2007. The file name is SaveAsPDFandXPS.exe and if you do a Google search for "Microsoft Save as PDF or XPS" (include the quotes) you should see the link within the first couple of search results.

You will need to validate your copy of Office, and then download & install the add-in. Once you have done that, to print an Access report as a PDF is as simple as can be:

Sub SaveAsPDF(MyReport As String, MyFile As String)
DoCmd.OutputTo acOutputReport, MyReport, acFormatPDF, MyFile
End Sub

Just supply the report name, and the desired full path & filename (which must end with ".PDF"), and the PDF will be created. Note that if the file exists, it will be over-written without warning.

Moving to Access 2007

Office 2007 has generally received a lukewarm response, but mainstream support for Office 2003 is set to end in April 2009, so remember the Boy Scout motto of "Be Prepared".

To date, I have had three customers that upgraded a single PC to 2007, while the rest of the users remained with 2003. Two have since downgraded back to 2003 due mostly to issues with Access 2007 compatibility. Some users prefer the new Ribbon interface as compared to the old-style menus and toolbars, but others, especially power users, feel lost. But I digress.

Upgrading an older-version Access database for use with 2007 is very much like what happened years ago when upgrading from the 16-bit Access 2.0 to the 32-bit versions such as Access 95/97 or 2000+. Many issues cannot be resolved using the new version - in some cases 2007 cannot open the database at all, so you will need to retain at least one PC with Access 2000-2003 in order to make the necessary changes.

There are a few things that are not supported, like using ENVIRON() expressions for default values in tables, but the most critical changes relate to the VBA environment, especially when using code-behind with forms and reports. Unlike the prior 32-bit versions, Access 2007 apparently examines all modules upon startup, so any code artifacts or compile errors can prevent opening the VB project, and thus the MDB cannot be opened. In fact, a 2000-2002 format file can compile successfully but then crash when opened using 2007. The only good news is that 2007 will tell you which module it can't load, so that gives you a starting point. But none of the VBA code will run until that module is fixed, and don't be surprised if there is more than one module that needs rework.

In a previous post, I described the workaround for that scenario - open the MDB with 2000-2003, export and delete the code module, compact & repair, then reopen with 2000-2003 and re-import the code module, compact & repair, and then it will probably work with 2007 as expected.

In the past, it has been possible to share a 2000-format MDB with users having Access 2000, 2002, and 2003 without conflict. In fact, I once had a single test PC with four versions of Access installed - 2.0, 97, 2000, and 2002 - but I would not attempt this with 2007.

The bottom line: I strongly recommmend that when moving to Access 2007, all users must have 2007 (only) installed to prevent corruption and/or data loss.