Microsoft 365 features that help users manage their subscriptions, account settings, and billing information.
If you cancel the plan, it will stay on until it expires. Then you won't be able to use the apps, but you can still use the web version for free via www.office.com.
However, you shouldn't have to cancel the app simply because you are using macOS version 12. You can run an older installer and it should work.
You will need to uninstall the current apps. And install the downgraded version (16.88) here: https://officecdn.microsoft.com/pr/C1297A47-86C4-4C1F-97FA-950631F94777/MacAutoupdate/Microsoft_365_and_Office_16.88.24081116_Installer.pkg?apid=ets