Contributors will not stick around
Hardly anyone will stick around. People contribute a lot, but few actually care for the "big picture" and stay around to work with the project for more than a short while to get their changes incorporated. You should more or less expect that the person who just brought you the most excellent bugfix will not be around anymore once you have pushed their patch.
Many contributors are just users who will fix a specific problem or an issue that they have and want fixed, and once that is done their mission in your project is done and they wander off again. All improvements are good.
Ideally, you not only want your contributors to stick around, you want them to participate and appreciate the project to a level so that you can promote them to become (co-) maintainers of the project.
A maintainer is usually the term used for participants in the project with enough powers to control certain things, like merging other people's changes.
My experience says that you will have better success in getting more maintainers if you (as an existing maintainer) ask those you consider being contenders, rather than waiting and hoping for them to ask.