So I'm looking at different options for a 8 or maybe 16 core setup.
I would like to stick with Intel -- something 5000x based to stay close to the reference MacPro design. 5400 seems viable too. With the major architectural changes in 10.6 I think it might be wise to stay close to the Intel platform. IOKit should provide some insulation from changes but who knows.
Going AMD is definitely cheaper but compatibility is always a factor. Apple's switch to NV chipsets definitely makes it a more appealing option. Some additional hardware support for other NV chipsets may trickle down. It looks like several of the NV AMD chipsets are compatible now. Another plus with AMD is I don't have to use more expensive ram -- just ECC DDR2. That cuts the price down significantly.
Total is probably $500 AMD, $1000 Intel.
I can find some quad socket AMD boards fairly cheap but they're huge. A case to fit one would be probably $300-$400 by itself.