The feature sets pretty much complement each other, and Yahoo’s excellent code standards (all their yahoo names start with Yahoo.*) means they could conceivably be used with each other.
Here’s my impressions so far.
Scriptaculous (I never know where to put those dots) uses prototype as the underlying library, and I’ve grown used to using $().
Overlap includes: autocomplete, DOM wrappers, sliders, sortable lists, and some fading effects. Of course both include browser-agnostic AJAX wrappers (or "connection", in YUI lingo). I haven’t compared the DOM wrappers yet so I won’t compare them.
The sortable lists on scriptaculous seem easier to use. Also, prototype has Try.these(), which helps with making the endless "trying" for different browsers a bit less ugly. Yahoo’s code shows a more traditional try/catch approach.
Yahoo on the other hand is big on widgets. Their Calendars and trees are awesome (and don’t exist on scriptaculous).
They also have a nicer animation API and an event framework that allows you to clean up events (to avoid memory leaks) and attach events to elements that don’t exist yet. Yahoo seems to rely a lot on getting elements by their ID, which may annoy some or require node-generating code to also generate unique ids (which may or may not be an annoyance to developers).
I’m going to try using this library on a project where I already have scriptaculous and I’ll add an update with my experience. I have just the place to put that cool date picker.
Update: Corrected myself on the issue of sortable lists. I had incorrectly stated that Yahoo didn’t have them. They do, they’re just a bit harder to find.