DTM E59. Real Time ML & Generative AI Edge Computing Platform - Varun, NimbleEdge

From the Deep Tech Musings Podcast - Get actionable and tactical insights to take your Deep Tech startup from 0 to 1 [Idea to Traction].

Listen now on - Spotify, Apple, Google

Varun Khare is a technology entrepreneur who currently serves as the Chief Executive Officer of NimbleEdge. Varun has worked on a wide variety of machine learning research problems across reputed institutions including UC Berkeley, Max Planck Institute, Indian Institute of Technology Kanpur, and OpenMined, an open source community focused on advancing privacy-preserving technologies. His pioneering research on federated learning at OpenMined culminated in the founding of NimbleEdge, an on-device machine learning platform for real-time personalization. With his work at NimbleEdge, Varun is helping enable some of the largest consumer apps globally to deliver cost-efficient, real-time personalization in a privacy-preserving manner.

On this episode we discuss,

1. Motivation to Start: What motivated Varun to start NimbleEdge?

2. NimbleEdge’s Scope: What is NimbleEdge’s scope of features & services?

3. Market Gap Identification: What was the gap in the market that led to the genesis of NimbleEdge?

4. NimbleEdge Solution: What is NimbleEdge's solution to this problem?

5. Technology Framework: What technologies are at play to make this possible?

6. Dream11: Walk through of the real-time ML use case with Dream11

7. Future Vision: GenAI at the Edge

Listen to the episode here,

or on the platforms below

Links & References:

Naveen Verma- Varun Khare (LinkedIn)

Encharge - nimbleedge.com, LinkedIn

Pronojit Saha, DTM Podcast - pronojitsaha (LinkedIn), @pronojits (Twitter)

1. What motivated Varun to start NimbleEdge?

• Varun's Background and Experience: Varun's professional journey included extensive experience in Android development, machine learning, and augmented reality, laying the technical foundation for his future endeavors.

• Involvement with Open Mind: He was actively involved with Open Mind, an open-source organization that emphasizes privacy-aware technologies, which shaped his understanding of data privacy and edge computing.

• Privacy Concerns: The core motivation stemmed from a desire to eliminate the need to send raw data to centralized servers by enabling machine learning models to be trained directly on devices, enhancing privacy.

• Introduction to Edge Computing: Varun was introduced to the concept of edge computing during his tenure at Open Mind, where he realized its potential to address privacy and data governance issues.

2. What is NimbleEdge’s scope of features & services?

• Definition and Scope of Federated Learning: Federated learning is a specific technique within the broader field of edge computing, focusing on training algorithms across decentralized devices while keeping data localized.

• Comprehensive Edge Computing: Edge computing encompasses a wider array of tasks, including data processing, model inference, and training, performed directly on devices, beyond just federated learning.

• Scope of NimbleEdge: NimbleEdge's platform covers all aspects of edge computing, providing a holistic approach rather than focusing solely on federated learning.

• Privacy Prioritization: Federated learning is employed particularly when privacy concerns are paramount, allowing collaborative model training without compromising user data security.

3. What was the gap in the market that led to the genesis of NimbleEdge?

• Challenges in Edge Computing Deployment: Varun identified significant difficulties in scaling edge computing and federated learning solutions across diverse device landscapes, recognizing a market gap.

• Meta's Device Testing Efforts: He noted how major companies like Meta invested substantial resources into testing various device capabilities, highlighting the complexity of handling device diversity.

• Device Capability Variability: The challenge of managing a wide spectrum of device capabilities, from low-end to high-end devices, was a significant barrier to widespread edge computing adoption.

• Unified Edge Platform Vision: Nimble Edge was envisioned as a unified platform to streamline and simplify the deployment of edge computing solutions across various devices, addressing the identified market gap.

4. How was the market landscape when NimbleEdge started?

• Google's and Chinese Companies' Efforts: At the time, companies like Google and several Chinese firms were exploring edge computing primarily for their internal use, rather than offering comprehensive external solutions.

• IoT Edge Computing Support: Established companies like Amazon and Microsoft had robust support for IoT edge computing, focusing on industrial and smart device applications.

• Mobile Edge Computing Challenges: Mobile edge computing was less developed, facing significant challenges due to the complexity and diversity of mobile device ecosystems.

• Focus on ML Runtimes: Existing solutions predominantly concentrated on machine learning runtimes, lacking comprehensive data processing and model management capabilities for edge devices.

5. What is NimbleEdge's solution to this problem?

• Fully Managed ML Platform: NimbleEdge offers a fully managed machine learning platform designed specifically for edge devices, handling end-to-end processes from data capture to model inference.

• Data and Feature Processing: The platform manages data capture and feature processing directly on devices, reducing reliance on cloud infrastructure and enhancing privacy.

• Initial Machine Learning Runtime: The platform initially started with a focus on providing a robust machine learning runtime for edge devices, ensuring efficient model execution.

• Lifecycle Decoupling: Over time, NimbleEdge evolved to decouple front-end and machine learning lifecycles, allowing ML teams to update models independently of app updates, streamlining deployment and maintenance.

6. How does a new mobile app engage with NimbleEdge?

• Enhancing User Experience: The primary objective is to enhance user experience by leveraging real-time data processing and predictions to understand and respond to user behavior dynamically.

• Minimizing Cloud Costs and Latency: By processing data on the device, NimbleEdge significantly reduces cloud costs and latency, providing a more efficient and responsive user experience.

• Challenges with Traditional Systems: Traditional cloud-based systems struggle with scalability and real-time processing, often resulting in higher costs and slower response times.

• Leveraging Device Compute Power: NimbleEdge taps into the compute power of modern devices, offering a scalable and cost-effective solution for real-time data processing and personalization.

7. What technologies are at play to make this possible?

• Improved Device Capabilities: Modern mobile devices have seen substantial improvements in compute and storage power, enabling more complex tasks to be handled locally.

• Significance of Mobile Devices: Mobile devices now account for 60% of internet traffic, underscoring their importance and the need for efficient edge computing solutions.

• Data Generation on Mobile: The vast amount of data generated by mobile users presents both a challenge and an opportunity for edge computing to process this data locally.

• Practical Edge Computing Solutions: The convergence of increased device capabilities and mobile data generation makes edge computing a practical and viable solution for modern applications.

8. Engagement with Dream11

• Handling Spiky Traffic: Dream11 faced significant challenges in managing spiky traffic and ensuring real-time personalization for its users, which were critical for their business.

• Quick Integration: NimbleEdge integrated with Dream11's systems in just five days, a stark contrast to the months typically required for cloud-based solutions.

• Real-Time Personalization: The platform enabled Dream11 to offer real-time personalized experiences to its users, improving engagement and satisfaction during high-traffic events.

• Enhanced User Engagement: Dream11 experienced significant benefits, including improved user engagement and satisfaction, thanks to the real-time capabilities provided by NimbleEdge.

9. What real-time ML use case did Dream11 implement?

• Personalized Contest Ordering: Dream11 utilized NimbleEdge to personalize the order of contests shown to users based on their interactions, enhancing user experience.

• Feature Extraction and Inference: NimbleEdge handled the extraction of relevant features and performed inference directly on devices, allowing real-time adaptation of the app.

• Collaboration with ML Team: Dream11's machine learning team developed the model, while NimbleEdge managed its deployment and execution on devices.

• Training and Inference Flexibility: NimbleEdge enables both training and inference to be conducted on devices, offering clients flexibility in their machine learning workflows.

10. Is it easy to integrate an existing stack with your platform?

• Platform-Agnostic Design: NimbleEdge is designed to be platform-agnostic, offering bindings for popular mobile development frameworks such as Android, iOS, React Native, and Flutter.

• Support for Major Technologies: The platform supports major technologies commonly used in mobile app development, facilitating seamless integration with existing tech stacks.

• Ease of Adoption: The design ensures that clients can easily adopt edge computing without extensive rework of their existing systems.

• Seamless Integration Process: NimbleEdge provides a straightforward integration process, making it accessible and convenient for developers to implement edge computing solutions.

11. What's now in the future for NimbleEdge?

• NimbleEdge is actively working on enabling generative AI (GenAI) applications on devices. This focus stems from recognizing the increasing challenges in productionizing GenAI on the cloud, which is generally harder than other models.

• Cloud-based GenAI faces difficulties such as the need for continuous streaming of token predictions or handling large files like audio and images, which aggravate existing cloud challenges.

• Given these complexities, enabling GenAI at the edge for various use cases and applications within mobile apps is crucial. NimbleEdge is concentrating its efforts on this area to streamline and enhance the deployment of GenAI solutions on edge devices.

• The emphasis is on overcoming the hurdles associated with cloud-based GenAI by leveraging edge computing capabilities, ensuring better performance, reduced latency, and more efficient data handling for mobile applications.

If you enjoyed this episode, please leave us a rating on Spotify, Apple, or wherever you listen to podcasts. It helps us reach more people who are interested in deep tech & grow the community.

Also, don’t forget to subscribe to Deep Tech Musings Podcast on pronojits.substack.com so you never miss an episode.

Thank you for listening! See you next time!