No, you don't have to use any specific toolset and you don't have to use any specific (physics) library either.
It's just a question of convenience, since things like Box2D will provide you other benefits as well, e.g. being able to do collision, physics simulation etc.
If you're not using tiles, you'll most likely want to write your own custom editor.
As an alternative, you could as well use tiled to draw rough outlines that are then interpolated by the game engine to create smooth surfaces (this could get tricky to edit!).
Creating smooth landscapes is actually pretty easy, but there are many things where you can optimize your algorithm.
Here's a short example of some basic "smooth landscape":
Your landscape or map is defined by several height points, in essence it's a 2D height map:
To fill the gaps, you can use any algorithm you want, e.g. splines or just linear interpolation:
If you worry about performance/complexity, you can use different algorithms for game logic (like collision detection) and rendering.
Now, once you've got your player character (or any other entity), you determine the closest height map point to the left and right of that entity. Based on that it's quite trivial to compute the actual ground height at that specific position:
For linear interpolation, it's possible to draw/calculate the triangle the character is standing on. Once you've determined the slope (deltay / deltax) it's easy to determine the ground height where the character stands.
Now, to prevent your character from falling through the ground, all you have to do is move him up as long as he's under the ground. This will also allow your character to walk over the terrain if you move him left/right.
There's just one issue left: Don't allow the character to climb too steep ground. There are multiple possible approaches, e.g. just check the slope. If it's too steep, don't allow the character to walk towards it anymore.
Of course, this is simplified a lot and this won't allow you to create caves, floating platforms, etc. However, those can be done as easy, you'll just need a more complex algorithm. Rather than checking a chain of points, you'll have to determine whether specific parts of your character overlap with map objects (= polygons).
Edit: Since I've had a few minutes, I've created a quick example utilizing jsFiddle (Source). Just make sure to click in the results window and you'll be able to run around using the left/right arrow keys.
