How do we perform database schema upgrade? Currently I am doing it manually by calling each of the upgrade_from_<from>_to_<to>.sql version files. Does calendarserver_upgrade take care of it? If not, is there an easier way of doing it?
Yes, calendarserver_upgrade has one job: to apply any necessary schema and data version upgrades. It is intended to be used immediately after upgrading to a newer version, before the service is started.
Currently for upgrading from 3.2 files to 5.1 db backend, I am just creating a new db. And the migration from 3.2 files backend to 5.1 db seems to work fine. Is there anything else that I should take care?
That sounds like a workable strategy for this upgrade delta. Going forward, calendarserver_upgrade should be all that is needed.