The effect of the mesmer is to make the target do exactly what you want. It's generally used to force people to give up information.
In the third book, Artemis Fowl, Juliet and Butler trick the fairies into believing that they are mesmerized. However they were acting, as all of them were wearing mirrored contact lenses.
To a human, the voice of the caster of the mesmer sounds like an entire choir singing in harmony. The sound is almost irresistible to a human. After the mesmer has been cast the human's eyes will look bloodshot and red, and therefore it is easy to tell if someone has had the mesmer used on them.
The eye contact needed for the mesmer can be obtained through a digital screen. When the mesmer was used to order a crowd to kill Butler and Juliet Butler the eye contact was made through a large screen.
By fairy law, a fairy can't use mesmer on another fairy. This law has been violated by Opal Koboi and Lt. Cudgeon. Although Leon Abbot (N'Zall) used mesmer, demons are not bound by fairy law, so he did not violate it.