     Subject: FYI: Access violation in IDPDOX32.DLL
        Date: Mon, 17 Mar 1997 11:39:36 +0100
        From: Hallvard Vassbotn <hallvard@falcon.no>
Organization: Falcon AS
         To:  76711.666@compuserve.com, lisch@tempest-sw.com, 
              Ralf.Quint@bon.netsurf.de, r.p.sterkenburg@dataweb.nl,
              70630.717@compuserve.com


Bug in          : BDE Paradox driver
BDE versions    : 3.12, 3.5 and 4.0 (from Delphi 3.0 beta)
Also in         : Paradox Engine for Windows, Paradox Engine for DOS
Description     : Access violation in IDPDOX32.DLL after adding about
                  100 MB to Blob file!
Found by        : Hallvard Vassbotn, hallvard@falcon.no, Reuters Norge AS
Severity        : Extremly serious


Background:
As the research and development department of Reuters in the Nordic contries,
we develop real-time and historical data systems for the financial markets.
One of the key components in this is our historical database of news items,
known as HistNews.

HistNews stores all stories from all news agencies covering the nordical
financial markets. It has news stories from 1989 until today and consists
currently of about 400.000 news items and about 800 new items are added
every day. These HistNews databases are replicated at each client site
(about 300) and are stored using the Paradox 4.0 database format. The
size of the database is about 385 MB (84 MB for the DB file, 300 MB for
the Blob file and 1 MB for the primary index).

The program that updates the Paradox tables run each night and is a
Dos-based protected mode program written using BP 7.0 and Paradox Engine
for DOS 3.01. Lately we have had problems in that the updating program
crash with a GP from time to time inside the Paradox Engine library.
This is a major headache as we must travel to the client site to replace
his faulty tables.

After spending days trying to re-create the problems in-house we finally
came to a sceniaro where we could constantly recreate the problem. This
happened on a local disk with no other potential conflicts with other
applications or users. The sceniaro is to let the updating program create
a new table on the local disk, then run in a loop in a batch file updating
the same set of news over and over again. The only difference is that we
set the date for wich the news should be added. This way we simulated
adding one years worth of data in a few hours.

Each day was updated using the exact same datafile and it added all the
days just fine until the middle of desember of the simulated year. Then
the blob file was about 130 MB and the program crashed with a GP giving
an address inside the PXISAM1 module of Paradox Engine for Dos 3.01
(pmode).

The next step was to actually step through the assembly code of Paradox
Engine using the Turbo Debugger. This quickly revealed that the crash
happened inside a call to the PXBlobOpenWrite function. The dump from
Turbo Debugger around the crashing instruction looked like this:

Turbo Debugger Log
PROTECTED CPU 80486
0B57:128E 45             inc    bp
0B57:128F 55             push   bp
0B57:1290 8BEC           mov    bp,sp
0B57:1292 83EC04         sub    sp,0004
0B57:1295 8B4604         mov    ax,[bp+04]
0B57:1298 BA0900         mov    dx,0009
0B57:129B F7EA           imul   dx
0B57:129D 8BD8           mov    bx,ax
0B57:129F 8B87AA29       mov    ax,[bx+29AA]
0B57:12A3 8B97A829       mov    dx,[bx+29A8]
0B57:12A7 8946FE         mov    [bp-02],ax
0B57:12AA 8956FC         mov    [bp-04],dx
0B57:12AD C45EFC         les    bx,[bp-04]
0B57:12B0 26803F01       cmp    es:byte ptr [bx],01         { <- GP
Here! }
0B57:12B4 750B           jne    12C1
0B57:12B6 8B56FE         mov    dx,[bp-02]
0B57:12B9 8B46FC         mov    ax,[bp-04]
0B57:12BC 050300         add    ax,0003
0B57:12BF EB0D           jmp    12CE
0B57:12C1 C45EFC         les    bx,[bp-04]
0B57:12C4 8B56FE         mov    dx,[bp-02]
0B57:12C7 8B46FC         mov    ax,[bp-04]
0B57:12CA 26034709       add    ax,es:[bx+09]
0B57:12CE EB00           jmp    12D0
0B57:12D0 8BE5           mov    sp,bp
0B57:12D2 5D             pop    bp
0B57:12D3 4D             dec    bp
0B57:12D4 C20200         ret    0002

In addition to this simple localization of the problem, we also tried to
find out _why_ the crash happened here. We did not find the exact solution,
but it seemed to have to do with Paradox Engine checking the wrong flag
before using a structure (i.e. it assumes the structure is valid when it
is infact not). One routine invalidates one flag, but another routine checks
another flag.


Knowing that Paradox Engine for DOS is a product that Borland does not longer
support, we decided to port the updating program to a 32-bit Delphi console
application that uses the BDE32. We reckoned that these bugs in Paradox
Engine probably (hopefully) was removed from the Paradox driver in BDE.
After spending almost a week converting the updating program to use the TTable
component to update the database, we found that it did work according to
the specs (i.e. the old 16-bit windows client applications could read
the database written by the new 32-bit update program).

Then we started the heavy testing again with excactly the same sceniaro
and datafiles that had crashed the 16-bit DOS protected mode update program.
We found that the new update program ran faster than the old one - but
it did still crash after updating about 9 months of news items.

Back in the debugger we found that the Access Violation is inside the
BDE Paradox driver, IDPDOX32.DLL. The address is 4CE0615D and the assmebly
code look like this:


Turbo Debugger Log
CPU Pentium Thread #215
:4CE06146 C20C00         ret    000C
:4CE06149 55             push   ebp
:4CE0614A 8BEC           mov    ebp,esp
:4CE0614C 0FBF450C       movsx  eax,word ptr [ebp+0C]
:4CE06150 8D14C0         lea    edx,[eax+8*eax]
:4CE06153 8B4D08         mov    ecx,[ebp+08]
:4CE06156 8B8411B9000000 mov    eax,[ecx+edx+000000B9]
:4CE0615D 803801         cmp    byte ptr [eax],01                {<- AV Here! }
:4CE06160 7505           jne    IDPDX32.4CE06167 (4CE06167)
:4CE06162 8D5003         lea    edx,[eax+03]
:4CE06165 EB06           jmp    IDPDX32.4CE0616D (4CE0616D)
:4CE06167 0FB75009       movzx  edx,word ptr [eax+09]
:4CE0616B 03D0           add    edx,eax
:4CE0616D 8BC2           mov    eax,edx
:4CE0616F 5D             pop    ebp
:4CE06170 C20800         ret    0008
:4CE06173 55             push   ebp
:4CE06174 8BEC           mov    ebp,esp
:4CE06176 53             push   ebx
:4CE06177 56             push   esi
:4CE06178 57             push   edi
:4CE06179 8B7508         mov    esi,[ebp+08]
:4CE0617C 8B5D14         mov    ebx,[ebp+14]
:4CE0617F 66837D1000     cmp    word ptr [ebp+10],0000
:4CE06184 7415           je     IDPDX32.4CE0619B (4CE0619B)
:4CE06186 668B03         mov    ax,[ebx]
:4CE06189 50             push   eax
:4CE0618A 56             push   esi
:4CE0618B E8A7FAFFFF     call   IDPDX32.4CE05C37

The observant will see that this is exactly the same logic as the 16-bit
code above - the only difference is that this is compiled as 32-bit code.
The conclusion is that the old bug from the Paradox Engine code is also
present in the newest BDE version.

It has been a long struggle to get where we are today and my attemts
to make a simplified program that exhibits this bug has been in vain.
The biggest problem in doing this is that the bug does not surface until
you have run the update program for 2-3 hours. Because of this I'm
forced to send you the executable version of my update program together
with the datafiles it needs for the updating and some batch files to
make it run in a loop.

I also submit the key units in the program so that you can see how
I do it. There also a simple program (PDOXBUG) that I wrote to try
to recreate the bug in a small example, but I have not made this
version crash. The bug probably only surfaces under certain
circumstances that has to do with the changing sizes of the blob fields.

So it seems that there is a set of conditions that must be true before
this bug is triggered. The Blob file must be of a certain size, the
blob fields added must be of a certain size and added in a certain
sequence etc. But this could occur in any product that uses BDE to update
Paradox tables containing Blob fields.

It is very urgent for Reuters that this bug is fixed as soon as possible
-
we have several hundred clients with databases that could corrupt any
day, and they do corrupt at the rate of 3-4 clients every week. If this
cannot be resolved in a reasonable amount of time, we will have to consider
our relationship with Borland and be forced to look elsewhere for
software development products.

Sincerly yours
Hallvard Vassbotn
Software Developer
Falcon R&D, Reuters Norge AS
hallvard@falcon.no
Phone: +47 22 00 57 33
Fax  : +47 22 42 50 32


Available upon request:
HNTEST.ZIP - The 32-bit update program with datafiles and batch files.
Unpack all files to a single directory. Run TTHN.BAT to test updating
an entire year. You should have at least 150 MB free diskspace. In our
testing this has crashed the BDE32 with a Access Violation at
4Ce0615D trying to read at 00000000. The batch file directs all output
to file called HNTEST32.LOG.

HNUPDSRC.ZIP - The source code for the updating program. Copyright
Falcon AS and Reuters Norge AS.

PDOXBUG.ZIP - A simplified application that will update a database
with a blob field. Run it from the PDOXLOOP.BAT file to let it
run "forever". We have _not_ been able to crash BDE32 with this program.


-- 
Hallvard Vassbotn
Falcon R&D
Reuters Norge AS

Document5, version 05 April 1997	page 4 of 5


