Juana Nakfour, Senior Software Engineer in the AI Center of Excellence for the office of the CTO at Red Hat, answers questions around the Red Hat's Open Data Hub below.
Watch the Open Source Community Project video.
Open Data Hub Video—Juana Nakfour
Hi Juana. For starters, why don’t you tell us about your project?
So, our project is an open source community project called Open Data Hub. It’s an end-to-end AI/ML platform.
What’s "end-to-end" mean, in this case?
¨End-to-end¨ means we provide all the tools for all the users of an AI/ML platform. From the data engineer to the data scientists to the DevOps and business intelligence users.
Gotcha. What’s it based on?
It's based on all open source projects such as Jupyterhub, Apache Spark, Seldon, Prometheus, Grafana and Argo. So there's a lot of flexibility in getting the community to innovate and contribute It is easy to use, because it's basically very easy to install and straightforward to use those tools. You can download it today, in a Red Hat OpenShift installation, from the catalog.
What do you think led to these projects happening in the first place?
I think behind a lot of challenges today, with regards to the data scientist and developers on AI/ML software engineering, is the fact that there are a lot of different concepts they have to be aware of. Open Data Hub makes it easier for them to just jump in and start writing their code that's AI/ML- specific, and not worry about where this is running, or what containers are running on. These projects help the community more than anything in AI/ML workflow development.
In your own words, can you explain the importance of being open source?
I like the word "open," meaning you can see every single line of code. And not only can you see it, but you can make improvements. You have the freedom to make improvements. That means you have this large group of people coming together, and that's where innovation happens. They all come together, discuss things, and try to fix things. That's where you see innovation happening for us.
For Red Hat and for Open Data Hub, that's our number one.
What are some other benefits that will come with being open source?
I think the open source part also plays into how flexible the platform is. From an Open Data Hub perspective, or any open source project, most of the time it's modular, so you can take components, and put components in according to your specifications. You pick tools and install them based on your individual requirements and needs.
Operators are the focus, I suppose.
Open Data Hub is a meta-Operator that has a lot of tools packaged together that can easily install an end-to-end AI/ML platform at once. Just the fact that they're modular and all in together, connected, means you can use module A, together with module P together with module E, which makes it easier for data scientists and engineers to develop faster.
These are very complex systems, though.
AI/ML is a complicated system. And there's a lot of modules that need to work together, from the beginning of data ingestion, to the middle of data science and data analysis, to the end of data model serving in DevOps and monitoring. It’s a complicated system, and we're putting it all together and providing it for users.
So like you just said, it sounds quite complex and complicated. How do you make sure that you avoid any glitches and make sure it's all running smoothly?
That was going to be my next question.
First, it's open source. This is community driven. So any issues you have, you can come to the community.
But, we have a monitoring system at the end of the deployment that I mentioned, which actually monitors how your models are performing and gives you feedback to see if there are any issues. And Openshift, as a platform itself, has a lot of monitoring tools that helps DevOps operators figure out where the problem is, where the issues are.
It has tools to restrict resource usage, to avoid issues with the platform. There are always balances. But the tools we have to resolve bugs are the tools everyone is using to track them, and every member of the team is in the trenches, writing code, solving bugs.
Final question: Personally, what is the most exciting part of this work?
I like to invent things. Especially when I can invent something that provides something new to the open source community. I guess just contributing back to the open source community makes me feel like I've added something to the open source world, and to users ́ experience. And to the whole AI/ML platform.