Some Herefordshire councillors are not pulling their weight in key meetings, a political leader in the county has claimed.