I've got calendarserver running under 10.7.4 and it works flawlessly except for external invitations. I've added a dedicated calendar user and setup imap/smtp settings per the documentation. 

unknown calendar user address: mailto:validuser@domain.com
error during PUT for mailto:validuser@domain.com: Unknown recipient

I also tried with addressbook disabled but get the same error.

Checking SMTP logs I see not activity.

Any help is most appreciated.

Regards,
Andy Knasinski
NRG Software, LLC.