"So I was going to say that maybe you didn't have enough Law to solve that problem, but it sounds like you have an artificial substitute which would be fine for something like this. Swear an oath that only binds you to action after you've heard everyone else in the village swear the same oath. Publicly generate random numbers. Three people picked by the random numbers, or however many it takes, sacrifice themselves to kill the adventurer."
"Put up a sign outside your village saying this is how your village does stuff. The adventurer reads the sign and goes somewhere else. Other people hear about what a great adventurer-free village that was and say 'Hey why don't we put up a sign like that too.'"