building with Scons & MSVC8 using PCH with PDB and /Zi

Some note concerning my current attempts to evaluate Scons 1.2 for my needs to replace my 'pure' Visual Studio solutions builds.

After trying to automatically convert my solutions to Scons scripts, which did not work at all, I decided to start from scratch.

I easily got the bare build and link done on one of my modules. And I tried to gradually introduce 'features' in my build : namely Precompiled header (PCH) support, and Debug information generation.

Adding both was easy, using the Scons man page as a reference, I used a construct like this one :

env['PCHSTOP'] = "precompiled.h"

env['PCH'] = env.PCH( os.path.join(builddir, 'precompiled.cpp') )[0]

env['PDB'] = os.path.join( builddir, "%s.pdb" % BASE_NAME )


Using this setup, it works fine out-of-the-box, but Scons msvc tool uses the /Z7 compiler flag to create the debug information. This has the effect to store debug informations in every built obj file, and according to Scons man page (at CCPDBFLAGS information), allow parallel builds to work fast & correctly.

This duplication of debug information lead my, simple, build directory to grow from about 33Mb (using my existing reference MSVC vcproj to build) to about 60Mb, almost 100% bigger...

I took a look at using /Zi instead of /Z7, to have a single PDB containing my debug information, and it took some to figure out how to have it work with PCH altogether.

According to Scons man page, all you have to do is ::

env['CCPDBFLAGS'] = ['${(PDB and "/Zi /Fd%s" % File(PDB)) or ""}']

Using this setup, I had two problems :

• the linker complained about LNK4203 errors, preventing debugging information to be linked.
• when doing a partial build of the project, the compiler complained about a C2859 error, indicating that the PDB file was not the one generated with the PCH, and instructing me to rebuild all.

In fact, what happens, is that Scons instructs the compiler to issue debug information in a given PDB file (say A.PDB) and also instructs the linker to output the debug information in A.PDB. My supposition is that the linker first starts to trash the PDB, and hence cannot find the debug information (LNK4203 errors). Afterwards, the A.PDB file is the one generated by the linker, and the compiler cannot use it anymore with the PCH...

I changed the PDB flags to instruct the compiler to use an intermediate PDB file :

env['CCPDBFLAGS']  = ['${(PDB and "/Fd%s_incremental.pdb /Zi" % File(PDB)) or ""}']

And now everything works fine.

A few performance tests on sequential file read methods under Windows

After reading Timothy Farrar's post on I/O thread scheduling, I thought that I made my latest performance tests on File I/O (especially read) too long ago.

I took some time to build a few no-brainer test functions, reading a file of about 1Gb, all read requests are 64Kb.

I essentially tested different a few variations :

• STDIO - standard stdio streams
• STD_S - standard stdio streams + 'sequential' microsoft extension (fopen(filename, "rbS"))
• WIN32 - Win32 API CreateFile using FILE_ATTRIBUTE_NORMAL
• WINNB - Win32 API CreateFile using FILE_FLAG_NO_BUFFERING
• MAPPB - Mapped file input (unrealistically mapping the whole file, and lazily using boost::iostreams::mapped_file)

As I'm not currently interested in latency, but more in brute-force throughput, I did not bother tested any async I/O.

On the same file, on the same hard drive (Seagate ST3250410AS), on the same machine, I ran the test on an XP32 and a Vista64.

To make it short, timings follow, it seems that :

• under XP32, using FILE_FLAG_NO_BUFFERING is worth the hassle. For lazy people, using Microsoft extension to fopen ("rbS" mode) looks like an easy good win.
• under Vista64, memory mapped file is the slowest approach, and all others compare almost equal, with a win for fopen in "rbS" mode.

I made quick tests on other machines, and the trends are the same.

XP32

Y:\>bench_disk_access_code.exe f:\ortho1.megatex
STDIO f:\ortho1.megatex : 33.35 Mb/sec. - 995754.00 Kb
WIN32 f:\ortho1.megatex : 33.57 Mb/sec. - 995754.00 Kb
STD_S f:\ortho1.megatex : 68.37 Mb/sec. - 995754.00 Kb
WINNB f:\ortho1.megatex : 87.09 Mb/sec. - 995754.00 Kb
MAPPD f:\ortho1.megatex : 79.23 Mb/sec. - 995754.00 Kb

Vista64

Y:\>bench_disk_access_code.exe f:\ortho1.megatex
STDIO f:\ortho1.megatex : 86.45 Mb/sec. - 995754.00 Kb
WIN32 f:\ortho1.megatex : 86.94 Mb/sec. - 995754.00 Kb
STD_S f:\ortho1.megatex : 88.42 Mb/sec. - 995754.00 Kb
WINNB f:\ortho1.megatex : 86.82 Mb/sec. - 995754.00 Kb
MAPPD f:\ortho1.megatex : 76.58 Mb/sec. - 995754.00 Kb