First of all, all the computers on your network need to talk to each other and share their various home folders or other user defined sharing locations. Very easy to do on the Mac side with Bonjour -- integration with Windows shares would be more tricky and probably require some Apple software on each machine to make the process easier. For security reasons we'd have to worry about authentication across machines. A simple wizard that went through every machine found in Bonjour and asked for credentials would solve this problem.
For data you would have a few pre-defined portals. Video, Music, Documents, etc. It should be easy enough to match these locations on a Windows machine (My Music, My Documents, etc)
Once a portal is created you check which machines (local & cloud) that you want to include. Color labels or icon branding would identify where the file actually was. A file unreachable would obviously be grayed out or otherwise noted as being offline.
So now you'd open your Music portal. You'd be presented with an interface similar to a stripped down iTunes. You could browse by album, artist, or go into CoverFlow view. This would pool all the music, on all your machines, and present them as one large collection. A context menu option would exist along the lines of 'Reveal Location' where you could open the actual share the file is located on.
Other context menu options might include "Copy local" - "Copy to iDisk" - "Sync"
Sync would copy the file to every machine and monitor the file for changes.
Copy local or, make available offline, would copy the file to a specific machine.
This whole system would be greatly enhanced by building into OSX a good, easy to setup, VPN solution. So when you're at work with your laptop your portals still function. Location awareness (either by GPS or manual selection) could offer an extra level of control. For example, when I'm at the office my extensive collection of marginally erotic 1920's pornography probably shouldn't be part of my portal.
The Pictures portal would be very similar to iPhoto -- if you wanted to edit the phones or do any serious work iPhoto itself would open.
If multiple versions of the same file exist locally and remote, portal prefers the local resource after checking for consistency.
Portals would replace the current open/save dialog. The dialogs would need to present the user with enough information to make this thing practical. For example, opening a 4GB HD iTunes rental over 3G probably isn't a good idea.
The whole concept would rely heavily on good peer-to-peer VPN (Hamachi style?) and iDisk. With that in mind, iDisk should be part of OSX. Perhaps 1 or 2GB for free with a reasonably priced upgrade option. I would personally be willing to spend quite a lot for cloud storage if it was integrated into everything I did seamlessly. This would be an excellent business model for Apple going forward.
Integration with future iPods and iPhones is obvious.
Integration with Windows should be a best effort sort of thing... or you know what? Just forget it entirely. Make it a Mac only feature. Trying to deal with Microsoft on something like this would be a nightmare. I am doubtful it would ever work 100% so declaring it as a 100% OSX exclusive feature is far more honest and sets the expectations for the end user.
It's an interesting idea. It has its share of problems but I think it's do-able and we're long overdue for a more modern, cloud aware, way of dealing with data. The emergence of wifi & cell dat, along with faster residential broadband, makes it a reality Apple could pull off today.