I owned two X5s (2003, 2007). In the first iterations of the X3 and X5 the X3 was too small and the X5 was just right. They have since made both vehicles bigger, so that the X3 is probably closer to just right, but still a little too small, the X5 is now too big (although the 2014s look a little smaller, but I have no facts to support that), and most importantly the X5 is really underpowered unless you get the V8.
The 3.0 was great for the original X5s, and then they made it bigger while keeping the same engine. They realized the problem and now sell them with a base '3.5' but that is sort of smoke and mirrors since it is still the 3.0L inline 6 with some turbo added. It still sucks though, bc a BMW should feel like driving a BMW and the 6 cylinder X5s do not. The V8 flies, but then you are looking at a starting MSRP of like 70K, at which point you might as well get a cayenne.
A Q5 is a good alternative that is right in between the X3 and X5 in size.