The issue you are facing is known as leaf roller caterpillar. Young leaves get stuck together and/or have numerous small holes with a chewed appearance. If the leaves are stuck together the small caterpillar can usually be found between the leaves.

Unless the infestation is heavy, neither the tree nor fruit will be a badly affected, but the tree may have an unsightly appearance. Pruning off the affected areas is quite often all that is necessary.