Boosting Embedded Analytics Performance with Semantic Layer

Empowering Open Source Communities with Data Insights

The Linux Foundation (LF) is a leading non-profit organization established in 2000 to support Linux development and open-source software projects. In addition to providing a neutral home where Linux kernel development can be fostered, the LF is dedicated to building sustainable ecosystems around open-source projects to accelerate technology development and encourage commercial adoption. With over 900 open-source projects as part of its portfolio, the LF plays a crucial role in the global open-source community, hosting major events like the Open Source Summit and KubeCon to foster collaboration and innovation.

Caption: Linux Foundation’s Insights tool - an open-source project analytics tool that empowers users with valuable data-driven insights with the help of a semantic layer

From Data Overload to Optimized Efficiency

Before partnering with Cube, the Linux Foundation faced challenges with their data stack's complexity and performance. Rudy Grigar, Director of Cloud Services at the Linux Foundation, explained, “We have lots of places where data lives at the Linux Foundation, anywhere from Google Sheets to Hubspot to Crowd.dev,” making it difficult to standardize and visualize data efficiently. Initial solutions were either too slow or too costly, prompting them to seek a more scalable and productive approach.

The first version of their Linux Foundatrion Insights tool, released five to six years ago, "could connect data sources, aggregate the data, and visualize it, but it was extremely expensive," Grigar added. While V1 could talk to Kafka and GitHub, it was not always accurate. This led them to search for a more reliable solution.

Their second attempt involved creating a data lake with centralized data governance and optimizations, but this approach struggled to keep up with performance demands and proved to be both slow and costly. Additionally, insights V2 relied heavily on computing data on the fly, which resulted in delayed performance despite accurate results.

Furthermore, the Linux Foundation's initial data infrastructure was not able to provide real-time insights due to the lack of advanced caching mechanisms and swift data aggregation processes. This led to scattered data depiction and inconsistent performance, making it challenging to deliver reliable insights to their diverse user base. “The performance aspect was difficult, and the question of ‘Is the data accurate?’ has always been a tough one,” Grigar noted.

Therefore, the Linux Foundation found themselves seeking a robust solution to streamline data management, embedded analytics, and visualization across multiple projects using a semantic layer; hence, they turned to Cube

Caption: Linux Foundation embedded analytics data stack

Making the Leap: Choosing Cube Cloud

The Linux Foundation had heard about Cube prior to their collaboration with Crowd.dev, but it was Crowd.dev's successful use of Cube Core that served as the tipping point for deeper evaluation. Grigar explained, “We had heard of Cube prior to Crowd.dev using it, and it just happened that Crowd was using it. We were discussing the architecture for Insights V3 and Cube was a key component of that,” particularly for its semantic layer capabilities. The big question was whether to run Cube in-house or use Cube Cloud.

The decision to choose Cube Cloud over Cube Core was influenced by several factors. First and foremost, the Linux Foundation wanted to avoid the operational overhead associated with hosting and maintaining Cube themselves. Grigar noted, “The big factor for me was whether we wanted to take on the responsibility and accountability of hosting the Cube stack, having to learn the ins and outs of scaling it, improving performance, and handling updates.” He further elaborated,

“I thought, do we want our team to be experts in hosting and running Cube, or do we want to focus on things more pertinent to the business foundation? That was the big driver for me.”

Additionally, Cube Cloud provided significant advantages in terms of integration and support. Grigar added, “Using Cube Cloud greatly benefits us. It requires minimal optimization time, ensures reliable deployments, and allows financial predictability. Moreover, it integrates with our GitHub workflow, providing significant advantages both financially and engineering-wise.”

The support from the Cube team also played a crucial role in their decision. Grigar highlighted, “We had hands-on support through the process, with group training sessions and continuous guidance on how to best use the tool. This support was invaluable in getting things off the ground and ensuring we were using Cube effectively.”

Ultimately, the combination of reduced operational overhead, seamless integration, financial predictability, and support led the Linux Foundation to choose Cube Cloud over Cube Core.

Effortless Onboarding: Implementing Cube Cloud

The implementation of Cube Cloud at the Linux Foundation was efficient and straightforward. Grigar shared, “It was quick to get started, and our development team was excited to use Cube.” As their data requirements grew more complicated, the Cube team provided vital support. “When our data started getting complex, the Cube team helped us better instrument our cubes and structure our aggregations.”

Cube's user-friendly design made it accessible for developers. Grigar noted, “Building a cube is pretty straightforward if you are familiar with any SQL language skills. It gets harder with more complex cubes, but it's much easier compared to the tools we were using previously.”

The swift implementation and support ensured that the Linux Foundation could quickly and easily employ Cube Cloud's capabilities to enhance their data-handling and information modeling processes through its semantic layer.

Revolutionizing Embedded Analytics and Data Management for Open-Source Projects

Implementing Cube Cloud significantly optimized the Linux Foundation's data processes by leveraging its advanced semantic layer. Grigar highlighted the role of pre-aggregations, stating, “Our applications would not be able to function without the pre-aggregation capabilities that Cube provides.” By harnessing embedded analytics, this automation allowed the Linux Foundation to maintain real-time data access and eliminate reliance on outdated data. “Pre-aggregations from Cube helped make our application real-time, so we don't have to rely on slow or stale data,” Grigar added.

Cube Cloud’s advanced data management features provided substantial benefits for the Linux Foundation's Open-Source Community Insights tool. “The caching layer and Cube Store have been instrumental in speeding up our data access and improving the user experience,” Grigar emphasized.

Additionally, the Linux Foundation heavily relied on the Cube REST API, which facilitated smooth integration with their access control solutions. Grigar explained, “We are able to plug in through our access control solution and integrate it with the Cube REST API.” The organization explored other API options, such as SQL and GraphQL, to further enrich their data-handling capabilities. “GraphQL has been on the table as an API, and there may be some places where GraphQL makes more sense for us. It’s nice to have the options for that,” Grigar noted.

The flexibility and range of API options available with Cube Cloud enabled the Linux Foundation to adapt their data operations strategies to meet diverse project requirements, ensuring efficient and streamlined operations across their extensive portfolio of open-source projects using embedded analytics.

Enhancing Developer Productivity with Real-Time Insights Across Multiple Dashboards

Furthermore, Cube Cloud offered developers at the Linux Foundation a more versatile approach to data governance using its semantic layer. Grigar explained, “Using Cube lets us pull from multiple data sources, standardize the visualization process, and cache through Cube Store, allowing for custom time ranges.” This enabled developers to quickly identify active development periods and drill down into specific date ranges.

The Linux Foundation's various dashboards, including Insights, Individual Dashboard, Control Center, and Organizational Dashboards, leverage Cube to render real-time data from diverse sources such as Google Sheets, HubSpot, and Crowd.dev. This integration ensures accurate, timely, and accessible data, facilitating better decision-making and insights for their projects. By unifying data management and visualization, Cube Cloud has empowered the Linux Foundation's developers to optimize their workflows and drive more impactful outcomes.

Comprehensive Benefits: DevOps and FinOps

Grigar also emphasized Cube’s profound impact on their DevOps, cloud operations, and engineering teams. "It's really helpful for us. My team doesn't have to sink a ton of time into optimizing; we can trust the deployments and rely on the service. Having the creators of the software manage the infrastructure is huge," Grigar explained.

Financial predictability has also been a significant advantage. Grigar noted, "Financially, we know what it's going to cost, which is super helpful. On the FinOps side, and on the engineering side." This predictability and integration have been powerful assets for the Linux Foundation.

Expanding Horizons: Future Innovations and AI Integration

Looking ahead, the Linux Foundation plans to expand its use of the Cube platform across more of its core products to ensure consistent data standards and streamline analytics across different applications. This expansion will include developing additional private or restricted access to data for internal teams, using Cube's access control capabilities.

Moreover, the Linux Foundation intends to further enhance its data cloud infrastructure by refining their dbt models and ensuring seamless integration with their expanding data sources. With Cube at the center of their data strategy, the Linux Foundation is poised to continue its leadership in the open-source community, driving innovation and productivity through advanced data analytics and AI integration.

Additionally, the Linux Foundation is committed to integrating AI capabilities into its platform. Grigar shared that there is an interest from their executive director to utilize AI further. "Cube would be a great place to leverage AI, so I'm interested to see if that would line up in the future," he said. They hope to explore the potential of using Cube's AI API to create more intuitive and engaging user experiences. For instance, implementing chatbots for board meetings to provide real-time insights on project contributions could be a significant Improvement.

By harnessing AI and Cube's robust data handling capabilities, the Linux Foundation aims to transform how users interact with and interpret data, paving the way for more informed decision-making and proactive project management. In doing so, the Linux Foundation continues to utilize Cube Cloud to drive open-source community insights, ensuring their projects remain at the forefront of technological innovation.

Caption: Linux Foundation Insights Embedded Dashboard