Plans to build up to six homes beside a Herefordshire village school and historic manor house have been knocked back.