Structure vs. Flexibility

I've been thinking a lot about the yin-yang relationship between structure and flexibility. In my education essay, I talked about how one of homeschooling's main strengths is flexibility while one of formal education's main strengths is structure. The divide between structure and flexibility is very deep. It can be seen in many places besides just education.

"The ABCs of how we learn," a literature review of education research, explains that there are two types of knowledge, which the authors call efficient knowledge and innovative knowledge. Efficient knowledge deals with how to refine existing tasks and solve recurring problems more efficiently. Innovative knowledge deals with how to form new processes and respond to novel problems.

"Primed to Perform," a book about high-performance cultures and their underlying motivations, describes tactical performance and adaptive performance. The authors define tactical performance as the ability to follow a plan and adaptive performance as the ability to diverge from a plan.

In "The Innovator's Dilemma," Clayton Christensen presents research about two kinds of innovation: sustaining innovation and disruptive innovation. As with the previous examples, these reflect structure and flexibility. Sustaining innovation gives incremental improvements to existing products while disruptive innovation creates new products based on early-stage technology.

A fundamental problem in machine learning is the balance between exploitation and exploration. The classic formulation of this problem is called the N-armed bandit problem. If you have N slot-machines (aka "bandits") with unknown payout distributions, what strategy should you use to maximize your profit? You need to find a balance between exploiting (playing on the slot machines that have given the highest payouts in the past) and exploring (trying new slot machines in the hope of finding some with higher payouts).

As a final example, there's a large debate in programming language design between static typing and dynamic typing. Statically-typed languages provide more structure while dynamically-typed languages provide more flexibility.

In all cases, the best solution involves working in the existing, known structure while retaining the flexibility to venture into new areas. This needs to be done in a way that includes and improves upon the strengths of both approaches. This is called a "win-win" solution by Stephen Covey, and it's different from compromise. In compromise you just find a half-way point that works well enough for both parties, but in win-win, you come up with a creative idea that's better than what either party had in mind to begin with.

So we should think about how to reconcile structure and flexibility in our endeavors. This is hard. In most cultures that I'm familiar with (and in my own life experiences), there seems to be a heavy bias towards structure. In education, public school is much more dominant that home school. "Primed to Perform" demonstrates quantitatively that most organizations try to optimize for tactical performance at the expense of adaptive performance. Statically-typed languages are more popular in industry as far as I can tell.

Why the bias? Is it just cultural, or is there some inherent cause? My hypothesis is that it's inherent. If you emphasize structure and not flexibility, you'll have solutions that work decently well, even though they aren't as good as they could be. If you emphasize flexibility over structure, you might have some cool ideas but you won't be able to turn them into concrete working solutions. Structure alone beats flexibility alone, so the bias towards structure is just evolutionary.

(As an example, imagine incompetent parents trying to home school their children vs. the standard practice of sending kids to public school. This is where the home school horror stories come from.)

We have plenty of structured solutions today, but we shouldn't stop here. The key to moving forward is learning how to introduce flexibility while keeping the benefits of structure. As implied above, the best route I think is to follow the progression from 1) nothing (no structure and no flexibility), to 2) structure only, and finally to 3) structure plus flexibility.

As we follow this progression, we can't let structure stifle flexibility. Structure brings much improvement when moving from nothing, so we come to view this structure as "validated." But after the structure is in place, flexible efforts to continue improvement often don't fall within the structure. If the structure is followed mindlessly, it will create a cage that restricts people from trying out new ideas. Flexible ideas can be seen as frivolous, bad ideas because they diverge from the already-validated structure.

I felt this acutely as a missionary. We had very detailed methods for goal-setting and planning. In our weekly planning sessions, we were supposed to set a lot of quantitative goals and talk about prescribed topics for how to plan around those numbers. The idea was to create an effective system that brings predictable results. But for the majority of my mission, it felt like the system was broken. In hindsight, I wish I had tried to be much more creative and flexible during weekly planning. I think we could have come up with great ideas for how to do missionary work in a way that met the needs of the locals.

So how do we avoid this pitfall? I think a few things are helpful. First is to remember win-win. When you find two opposing ideas, the truth usually lies in a combination of the two—not one or the other. With static vs. dynamic typing, I believe the answer is in some combination of the two ideas. (Clojure.spec is a good step in that direction, though it's not all the way there yet). Remembering win-win will make sure you seek for the real truth instead of getting stuck on one side of a useless debate.

Second is a healthy disrespect for conventional thinking. This will help you cultivate potentially good ideas which otherwise would be dismissed. One of Einstein's greatest strengths was that he was perfectly comfortable challenging the accepted wisdom of the scientific community. This allowed him to develop his theories of relativity which broke science free from the model of Newtonian mechanics that had governed physics for hundreds of years. Although it would be impractical to never rely on society's accrued knowledge, we should look for ideas that are ready to be challenged. No idea should be considered free from the need of further validation.

Third is a habit of experimentation. Any idea should be allowed as a hypothesis, but then it needs to be tested. Existing structure shouldn't prevent you from experimenting on new ideas, but you also shouldn't accept new ideas before they've been validated. My personal goal-setting and planning system has undergone much experimentation over the past three years. This is the process by which a lot of potentially good ideas become a few actually good ideas.

This is the key to combining structure and flexibility, and it's basically just the scientific method. It's an effective pattern for discovering truth in all areas, not just what we call science. There's more work to be done incorporating it into other parts of our lives.