Ran into another idiotic bug. "Too many vassals" issue can only go away if you give away your personal holdings. Creating new duchies to consolidate multiple counts under a single ruler doesn't solve the problem.
Also, from what I've been reading, embracing feudalism as a tribal ruler utterly gimps your nation as all your buildings go fuck themselves without being replaced by low tier feudal buildings as it was the case in CKII. In the entire realm. So not only will all your levies disappear into thin air, but you won't even have the money to maintain even your new gutted army.