Wiki and Open Data

Boulder & Eldorado Canyon, Colorado 39.92943, -105.33166 Eldorado Canyon State Park, Colorado 40.00442, -105.26706 University of Colorado, Boulder BOUNDARY 40.01, -105.26706

# Some background

We build systems of rapidly growing complexity. Experience troubleshooting failures in immensely complex systems often makes me wonder how any of these things work at all. Individual components seem so fragile. Entanglements between systems seem so pervasive. Unknown relationships and dependencies lurk in shadow. Predicting specific failures is impossible. Yet it is not impressive to predict that there will be catastrophic failures, cascading failures, unintended consequences.

Ward shows an example of how to explore relationships in a complex, rapidly growing system to discover risks to mitigate or identify opportunities for improvement.

**Investment**

It takes time and energy to curate data streams. And more time and energy to find common language for associating different sources of data. These efforts appear fruitless at first. But as data sets are groomed and associated and related, the insights to be found exploring relationships cannot be gained by other means.

**Exploration**

It takes time and energy to explore new territory. An organization or community must devote resources to the inquiry itself. El Dorado enables the explorers to capture and save and even share interesting queries and interesting results.

But these are explorations into the unknown. They cannot be planned. They require a different sort of agility and different acceptance criteria than the typical development story. Maybe they more closely resemble market research.

**How to persuade?**

Time spent exploring also comes with the opportunity cost of features that could have been built with that time and those resources instead of exploring. And Sales and Marketing teams are professional persuaders. What Development or Operations team can compete for the priorities of the Business Leaders.

Specific deals that were lost for lack of specific features? Insatiable demands from the sales pipeline. We promise the alternative of unknown riches of El Dorado? Or cures for diseases we don't know we have?

**Private and Confidential**

New Relic's investment in El Dorado (and likely other systems) empowers them to embrace team autonomy, and enables them to study their own emergence. But publishing the results would give up intellectual property and might also violate agreements with their customers.

How can I create something similar to deeply learn how to explore emergence in distributed systems and distributed teams? How can these lessons be shared in depth?

# Open Data

These words mean different things to different audiences. My friend Alex evangelizes a specific, and different sense of open data at University of Colorado. This is semantic web stuff... aiming to get researchers to open and link their data to empower greater collaboration.

These offices of the City of Boulder and the State of Colorado speak about open data in the sense of empowering businesses and citizens with data already collected and curated by various parts of government. I don't hear anything about triples or graph queries.

But there is an interesting opportunity in these data sets. They're public data. We could leverage the public data to begin building something like New Relic's El Dorado that is unencumbered by non-disclosure.

# Collaborations

I already mentioned my friend Alex at CU. It's a different notion of open data, but there are graphs to be explored in semantic web triple stores, and there are graphs to be explored in El Dorado. And overlapping goals. Potential for something interesting to emerge from collaboration.

Another friend, Brad, helps small- to medium-sized businesses identify savings in electricity and gas by monitoring and analyzing and re-shaping time series data into actionable reports. The savings outweigh the cost of monitoring and analysis. His could be a specific business benefiting from the open data (maybe) or (more likely) benefiting from tools and processes that facilitate mash-ups of apparently-but-not-actually unrelated data.

blockchain or merkle tree offer ways to formalize, publish and track the data supply chain from sources. The holochain community already working with David and wiki may be interested to help. The Boulder blockchain enthusiasts may also be interested to help.

Duke introduced me to Cristos at CSU who researches Named Data Networking. Seems like there are some great privacy stories to tell in connecting dots between NDN and open data.

I have a long interest in applying computing to education. There may be collaborations to teach data wrangling. My former mentor, Mark, directs the ATLAS Institute at CU. His friend directs the Craft Computing Lab for the CS dept. Mark in particular helped some former students with SBIR grant to bootstrap Modular Robotics. SBIR might be a useful funding mechanism. I also worked with Jeff who is now a Lead Curriculum Developer for Galvanize.