Start with what's observable, then move to outcomes
When I worked as a training measurement consultant, the one kind of training that was always assumed to be UNmeasurable was leadership training. Companies would kind-of accept defeat at the beginning of our conversation, assuming that it wasn't possible even though they somehow knew they had to spend money on it.
The reason leadership effectiveness is hard to measure is that it's just too darned vague! I could ramble on here about everything being measurable and blah, blah, blah...who has time for that. Let's get tactical. How does good leadership show up in an organization?
People - This is the one usually called out and it's a good one. But what about people? Do you measure retention of tenured employees? Retention of brand new employees? What about employee engagement scores or even exit interview feedback? All of these have varying degrees of precision in measuring people leadership, but all give measurable insight. The other cool people measure (that I like) is mobility. How often does a leader move people up through his organization to get promoted?
Innovation - How many new ideas come out of this leader's team? How often does this leader introduce new ways to do things? These are not rhetorical questions. Which is better, 10 new ideas, 6 of which fail, or 3 really great ideas? Both! Don't get caught up in what innovation means, but pay attention to (or just count) how often your leaders are pushing the envelope.
Budget - Yep, the leader whose org is consistently at or below her budget is likely to have good budgetary leadership skills. So what are her observable behaviors? Checks and balances on who gets to spend what? Early year prioritization of big spends? Simply being able to read her financial reports? Each of these is observable and is going to be a predictor of budgetary leadership success.
So those are just three examples of how good leadership shows up in an organization and the behaviors those leaders engage in to drive those behaviors. There are more, though. What does good leadership look like in your org?