A good way is to keep several sets of resources, each set for a single frame. Each set should contain a command buffer (or more), a fence and two semaphores.
Now You record a command buffer (or more), potentially with ONE_TIME_SUBMIT flag, and submit it to a queue. During submission You wait for "image acquired" semaphore that was provided to vkAcquireNextImageKHR() function. You also provide the second semaphore and the fence to be signaled. Semaphore is used for vkQueuePresentKHR() function.
For the next frame You take another set of the above resources until You run out of all sets. Then You just take the set that is the oldest (least recently used). You check if the fence from this set is already signaled. If it is, You record the command buffer and perform all the usual job. If the fence is not signaled, You wait for it. But if You have 3 or 4 such sets there is a small chance that the fence isn't already signaled.
You can't re-record a command buffer while it still being processed, so You need to wait. But this approach lets minimize waiting. You don't need to use functions like vkDeviceWaitIdle() and thus stalling graphics hardware pipeline. It's also easier to balance memory consumption, input lag and performance. The more sets, the less wait and better performance, but more memory is used and potentially You increase input lag. Typically 2, 3, maybe 4, sets should be enough.
Example of such approach is described in the Vulkan Cookbook and presented on Github:
Increasing the performance through increasing the number of separately rendered frames