
.png?auto=compress,webp&upscale=true&width=610&height=488&name=hubspot%20covers%20(6).png)
In the world of data mesh, data products are key. But what exactly is a data product and how do you approach the functional analysis of such a data product? You will discover it in this blog post.
Data Product?
The basic idea of a data mesh consists of 2 parts:
- Increasing the accessibility, availability and usability of data for business users.
- Reducing dependencies between data teams.
We base our approach on the principles of domain-oriented ownership, federated computational governance, self-service data platforms and product thinking. Particularly the latter is crucial in understanding and developing a data product. We aim to consider and shape data as a reusable product, enabling it to be utilised in various ways, thus maximising its value.
A data product is an autonomous, read-optimised, standardised data unit containing at least one dataset (Domain Dataset), created for satisfying user needs
— Majchrzak Jacek , Author at Data Mesh in Action
It is a logical unit that encompasses all components required to process and store domain-specific data for various use cases, such as data analytics, and makes it accessible to other teams through 'output ports'. A data product also has its own independent lifecycle and management structures. In essence, you can compare a data product to a microservice, but designed for analytical data.
Data products connect to sources through input ports, such as operational systems, data platforms, or other data products, and perform specific operations on the data, such as transformations, calculations, data anonymisation, and more.
Developing a data product involves addressing various aspects, including defining input and output ports, data cleaning, transformations, field mapping, GDPR compliance, and so on. Therefore, a thorough analysis is crucial.
But, how do you start such a data product analysis?
Step-by-step Guide for Data Product Analysis
A structured analysis approach ensures the best results. For data products, we rely on the Data Product Canvas, which we pair with a useful checklist.
The canvas is a visual representation that simplifies the display of the various critical components of your data product analysis. The checklist ensures that you don't overlook anything.
Data Product Canvas
With the Data Product Canvas, we ensure a consistent process for designing a data product within an organisation. The canvas succinctly outlines the aspects you need to consider during your analysis.
Using this canvas, you can engage various stakeholders of the data product. This collaborative effort leads to the desired result.
Based on our experience with this canvas, we recommend filling it out in a specific sequence. It's best to start with specifying the data product. This way, you immediately capture all descriptive data, ensure that all stakeholders are well-informed, and clarify the data product's purpose. Next, move on to the output ports since the data product's stakeholders often have a good understanding of the data they require. You can compare them with end-users of an application who benefit from a good user experience. Afterwards, address the input ports. Based on feedback from data consumers, you will identify the input sources capable of providing the necessary data. Finally, conclude with the data product's design. In this phase, input and output converge, and you aim to come up with a logical approach to transform input into the desired output.
Checklist
1. Data Product Specification
Make sure to collect all the descriptive data for your data product. This includes:
- Domain name
- Data product name
- Ownership: who manages the data product: The technical team's contact information, The technical team's contact information, The data product's expiration date)
- Security: Data product security is crucial: it determines how and by whom the data product can be used. Specify the license or 'terms of use' associated with the data product.
- Update frequency: The frequency determines how often the data product is updated. It's a good idea to also indicate how the updates are performed (e.g., full load, incremental, etc.).
- Data Consumers: A data product's data consumers are applications or other data products that will use your data product's output.
- Use cases: Use cases describe your data product's purpose. They provide more information about the data product's needs and rationale.
2. Terminology
In many companies and projects, confusion often arises regarding the meaning of certain concepts, terms, or words. Ubiquitous language, a Domain Driven Design principle, seeks to provide a solution by aiming for a vocabulary that is shared and clearly understood by all stakeholders.
To avoid speech confusion and interpretation differences when developing and analysing a data product, it's crucial to pay attention to ubiquitous language. Therefore, be sure to include definitions of terms that are relevant within the context of the data product. References to other glossaries, wikis, etc., are also welcome.
3. Output Ports
Output ports determine the format and protocol in which data is made available to your data consumers. Discuss with your data consumers (and other stakeholders) the format in which they prefer to consume data. Examples of output port types include analytical data, blob stores, Linked Data, etc.
4. Input Ports
Input ports specify the format and method by which source data can be read. By reaching out to the owners of your source systems (or source data products), you can discover the available options.
Be sure to mention the type of input port (e.g., API, database, file, etc.), and also note which tables of the source system the required data can be extracted from.
It's also very useful to include a visual representation of the source system's domain at this stage.
5. Data Product Design
This is the final and perhaps the most critical step in a data product's analysis. In this step, you will think about the logic of transformations within the data product. Consider:
- Identification of desired input data/fields
- Identification of desired output data/fields
- Constraints on data input (e.g. not all information is available in all cases)
- Data cleaning
- Data transformations
- Input field mapping
- Output field mapping
- Data enrichment
- Data anonymisation (e.g., in compliance with GDPR regulations)
- Data calculations
In the practical example below, we address several of the above topics. It illustrates how source fields are mapped and how to handle them effectively. Additionally, certain fields are calculated based on input data (e.g., location).
In Conclusion
Analysing data products involves many aspects. However, with a well-structured approach, a visual framework, and a simple checklist, you can complete this task with ease.

What others have also read


Our Business expert in Energy Utilities Tom Claus headed to Energy Mission, and came back with some new insights… and a wake-up call for companies and (government) agencies.
Read more

Enlit is Europe's largest event on energy transition. Tom Claus and Sven Sambaer went on behalf of ACA Group to meet up with customers and partners, and to keep their eyes open for new ideas and the latest trends. A report on the energy trilemma, the car battery as guardian of the balance in the distribution network, and smart metering 2.0.
Read more

November 30, 2023 marked a highly anticipated day for numerous ACA employees. Because on Ship-IT Day, nine teams of ACA team members, whether or not supplemented with customer experts, delved into creating inventive solutions for customer challenges or for ACA Group itself. The hackathon proved to be both inspiring and productive, with at the end a deserved winner! The atmosphere in the ACA office in Hasselt was sizzling right from the early start. Eight out of the nine project teams were stationed here. During the coffee cake breakfast, you immediately felt that it was going to be an extraordinary day. There was a palpable sense of excitement among the project team members , as well as a desire to tackle the complex challenges ahead. 9 innovative projects for internal and external challenges 🚀 After breakfast, the eight project teams swarmed to their working habitat for the day. The ninth team competed in the ACA office in Leuven. We list the teams here: Chatbot course integration in customer portal System integration tests in a CI/CD pipeline Onboarding portal/platform including gamification Automatic dubbing, transcription and summary of conversations publiq film offering data import via ML SMOCS, Low level mock management system Composable data processing architecture Virtual employees Automated invoicing If you want to know more about the scope of the different project teams, read our first blog article Ship-IT Day 2023: all projects at a glance . Sensing the atmosphere in the teams Right before noon, we wondered how the teams had started and how their work was evolving. And so we went to take a quick look... 👀 1. Chatbot course integration in customer portal “After a short kick-off meeting with the customer, we divided the tasks and got to work straight away,” says Bernd Van Velsen. “The atmosphere is great and at the end of the day, we hope to present a result that will inspire the customer . In the best case, we will soon be able to use AI tools in a real customer project with the aim of making more optimal use of the customer's many data.” “The Ship-IT Day is an annual tradition that I like to participate in,” says Bernd. “Not only because it is great to collaborate with colleagues from other departments, but also because it is super educational.” 2. System integration tests in a CI/CD pipeline “We want to demonstrate that we can perform click tests in the frontend in an existing environment and verify whether everything works together properly,” says Stef Noten. “We can currently run the necessary tests locally, so we are good on schedule. The next step is to also make this work in our build pipeline. At the end of the day, we hope we will be able to run the tests either manually or scheduled on the latest version of the backend and frontend .” 3. Onboarding portal/platform including gamification The members of this project team all started at ACA fairly recently. And that is exactly what brought them together, because their goal was to develop a platform that makes the onboarding process for new employees more efficient and fun . Dieter Vennekens shared his enthusiasm with us, stating, "We kicked off with a brainstorming session to define the platform's requirements and goals. Subsequently, we reviewed these with the key users to ensure the final product aligns with their expectations. Our aim is to establish the basic structure before lunch, allowing us to focus on development and styling intensively in the afternoon. By the day's end, our objective is to unveil a functional prototype. This project serves as an opportunity to showcase the capabilities of Low-Code .” 4. Automatic dubbing, transcription and summary of conversations Upon entering their meeting room, we found the project team engrossed in their work, and Katrien Gistelinck provided a concise explanation for their business. "Our project is essentially divided into two aspects. Firstly, we aim to develop an automatic transcription and summary of a conversation . Concurrently, we are working on the live dubbing of a conversation, although we're uncertain about the feasibility of the latter within the day. It might be a tad ambitious, but we are determined to give it a try." She continued, "This morning, our focus was on defining the user flow and selecting the tools we'll utilize. Currently, multiple tasks are progressing simultaneously, addressing both the UI and backend components." 5. Publiq film offering data import via ML Comprising six publiq employees and three from ACA, this team engaged in an introductory round followed by a discussion of the project approach at the whiteboard. They then allocated tasks among themselves. Peter Jans mentioned, "Everyone is diligently working on their assigned tasks, and we maintain continuous communication. The atmosphere is positive, and we even took a group photo! Collaborating with the customer on a solution to a specific challenge for an entire day is energizing. " "At the close of the day, our objective is to present a functional demo showcasing the AI and ML (Machine Learning) processing of an email attachment, followed by the upload of the data to the UIT database. The outcome should be accessible on uitinvlaanderen.be ." Peter adds optimistically, "We're aiming for the win." That's the spirit, Peter! 6. SMOCS, Low level mock management system Upon our arrival, the SMOCS team was deeply engrossed in their discussions, making us hesitant to interrupt. Eventually, they graciously took the time to address our questions, and the atmosphere was undoubtedly positive. "We initiated the process with a brief brainstorming session at the whiteboard. After establishing our priorities, we allocated tasks accordingly. Currently, we are on track with our schedule: the design phase is largely completed, and substantial progress has been made with the API. We conduct a status check every hour, making adjustments as needed," they shared. "By the end of the day, our aim is to showcase an initial version of SMOCS , complete with a dashboard offering a comprehensive overview of the sent requests along with associated responses that we can adjust. Additionally, we have high hopes that the customized response will also show up in the end-user application." 7. Composable data processing architecture This project team aims to establish a basic architecture applicable to similar projects often centered around data collection and processing. Currently, customers typically start projects from scratch, while many building blocks could be reused via platform engineering and composable data. “Although time flies very quickly, we have already collected a lot of good ideas,” says Christopher Scheerlinck. “What do we want to present later? A very complex scheme that no one understands (laughs). No, we aspire to showcase our concepts for realizing a reusable architecture , which we can later pitch to the customer. Given that we can't provide a demo akin to other teams, we've already come to terms with the likelihood of securing second place!" 8. Virtual employees This team may have been the smallest of them all, but a lot of work had already been done just before noon. “This morning we first had a short meeting with the customer to discuss their expectations,” Remco Goyvaerts explains. “We then identified the priority tasks and both of us quickly got to work. The goal is to develop a virtual colleague who can be fed with new information based on AI and ML . This virtual colleague can help new employees find certain information without having to disturb other employees. I am sure that we will be able to show something beautiful, so at the moment the stress is well under control.” Chatbot technology is becoming more and more popular. Remco sees this Ship-IT project as the ideal opportunity to learn more about applications with long-term memory. “The Ship-It Day is a fantastic initiative,” says Remco. “It's wonderful to have the opportunity to break away from the routine work structure and explore innovative ideas.” 9. Automated invoicing The client involved in this project handles 50,000 invoices annually in various languages. The objective is to extract accurate information from these invoices, translate it into the appropriate language, and convert it into a format easily manageable for the customer . “Although we started quite late, we have already made great progress,” notes Bram Meerten. "We can already send the invoice to Azure, which extracts the necessary data reasonably well. Subsequently, we transmit that data to ChatGPT, yielding great results. Our focus now is on visualizing it in a frontend. The next phase involves implementing additional checks and solutions for line information that isn't processed correctly." Bram expresses enthusiasm for the Ship-IT Day concept, stating, "It's fun to start from scratch in the morning and present a functional solution at the end of the day. While it may not be finished to perfection, it will certainly be a nice prototype." And the winner is …. 🏆 At 5 p.m., the moment had arrived... Each team had the opportunity to showcase their accomplishments in a 5-minute pitch, followed by a voting session where everyone present could choose their favorite. All teams successfully presented a functional prototype addressing their customer's challenges. While the SMOCS team may not have managed to visualize their solution, they introduced additional business ideas with the SMOCintosh and the SMOCS-to-go food concept. However, these ideas fell just short of securing victory. In a thrilling final showdown, the team working on the onboarding platform for ACA came out as the winners! Under the name NACA (New at ACA), they presented an impressive prototype of the onboarding platform, where employees gradually build a rocket while progressing through their onboarding journey. Not only was the functionality noteworthy, but the user interface also received high praise. Congratulations to the well-deserving winners! Enjoy your shopping and dinner vouchers. 🤩 See you next year!
Read moreWant to dive deeper into this topic?
Get in touch with our experts today. They are happy to help!

Want to dive deeper into this topic?
Get in touch with our experts today. They are happy to help!

Want to dive deeper into this topic?
Get in touch with our experts today. They are happy to help!

Want to dive deeper into this topic?
Get in touch with our experts today. They are happy to help!

