There really is no concrete definition for either term (or for the related term "framework," for that matter). There are hundreds of different ways of interpreting the varied subtle distinctions in software packages and assigning them classifications.
I personally view a "library" as a thing that provides a relatively small and directed set of functionality, whereas a "framework" provides more or broader functionality, and an "engine" provides the same (or more) and typically has a set of associated tools that come with it.
I have seen enough of other people's personal interpretations of the same terms to feel safe suggesting that an "engine" provides more, or higher level, functionality than a "library" typically does... but where you draw that line for your own software is up to you.