dot points on build page extensions

I have been thinking about making some extensions to the build page.

Raw Data

• Keep raw_data to process at any time. No need to discount old data collected from buggy analysis.

Profiling

• Use function wrappers, that log profiling of each test and multiple calls.


• -p|--profile command line mode

Tests

• Use subprocess mode by default for run_tests.py


• Web interface for ticketing off tests

Build information

• Post compiler version


• Post complete Setup file


• Post complete build output


• Post complete test output


• Python sys.path


• Environment variables


• As much as possible, unprocessed for archives

Machine information

• Processor speed


• CDRom availability


• etc, etc.

Breaking up tests

Should the tests fail if a machine doesn't have a CD drive (assuming stubs were filled out) for example?

Should tests that require Numeric or NumPy fail if neither available?

There are some classes of tests that it seems to make sense to split apart from the main "base" group of tests. What should be the "base" group of tests to automate with the run_tests.py test runner?

What about tests that require human verification? For the build page a "base" group of tests should be specified.

What should be the requirements for machines sending results to the build page? Numeric, Numpy? win32 extensions on windows? A CD rom drive? 32 bit color display?