Contact person: AR Nail App Enhancement & Advanced Features2

Phone:Show

Email:Show

Location: Johannesburg, South Africa

Budget: Recommended by industry experts

Time to start: As soon as possible

Project description:
"Reposting due to timewasters

Only bid if you can show you have experince in flutter/AR and ML/computer vision for the nail models etc

AR Nail Try-On App with Custom Nail Segmentation Models and 3D Overlays
Project Overview:

We are building an AR Nail Try-On app using Flutter and MediaPipe, which allows users to apply virtual nail polish in real-time based on hand tracking. The app already fetches color data from our API and provides a basic color selection. However, we need help applying the selected colors correctly to the nails and implementing high-quality nail segmentation models that can handle accurate nail recognition, mesh creation, and 3D overlays. Eventually, we would like to develop custom models for more advanced features, including meshing and realistic 3D curves for nail overlays.

Scope of Work:
1. Nail Color Application:

Objective: Ensure that the selected color (from the API) is properly applied to the nails on the AR model.

Tasks:

Integrate the selected color (fetched from the API) to be applied to the nail segmentation in real-time.

Make sure the color is applied accurately to the nail area based on hand tracking provided by MediaPipe.

Handle different skin tones, lighting conditions, and finger movements to ensure the color stays consistent and natural-looking.

2. Nail Segmentation Model Implementation:

Objective: Implement or enhance the existing nail segmentation model to accurately identify nails in real-time and apply virtual polish.

Tasks:

Integrate MediaPipe’s existing hand tracking model to provide accurate hand landmarks.

Use or develop high-quality segmentation models that can differentiate the nails from the surrounding skin and accurately track each nail’s position and shape.

Fine-tune the segmentation models to work in different lighting conditions, skin tones, and hand positions.

Explore and integrate additional models (e.g., deep learning models for nail segmentation) that will improve accuracy and performance.

3. Advanced Mesh Generation and 3D Nail Overlays:

Objective: Enhance the AR nail try-on experience by adding realistic 3D nail overlays and curves using advanced mesh generation and 3D modeling techniques.

Tasks:

Create 3D models of nails that can be applied over the detected nails in the hand.

Implement mesh generation for nails, which will allow for smoother and more realistic overlays.

Develop 3D curving overlays that wrap around the natural curvature of the nails for a more realistic look.

Ensure the nail polish application is dynamic, adjusting to user gestures (e.g., finger movement, rotation).

4. Custom Nail Segmentation Model (Future Development):

Objective: Build a custom nail segmentation model tailored to the specific needs of the app and its users.

Tasks:

Design and develop a custom neural network or machine learning model for accurate and precise nail segmentation.

Implement a solution for meshing nails and applying 3D curving effects in real-time based on tracked hands.

Train the model on a diverse dataset of hands with different skin tones, nail types, and angles to ensure robustness.

Optimize the model to work efficiently on mobile devices while maintaining high accuracy and low latency.

5. Integration with Flutter and MediaPipe:

Objective: Seamlessly integrate the segmentation and color application features into the existing Flutter app.

Tasks:

Work with Flutter’s platform channels to interface between the Flutter app and native code for MediaPipe and the segmentation models.

Ensure that color application, nail segmentation, and 3D mesh overlays work fluidly together without lag or performance issues.

Test the implementation on various Android and iOS devices to ensure cross-platform compatibility and performance.

6. Performance and Optimization:

Objective: Optimize the app for smooth and fast AR interactions.

Tasks:

Minimize the latency between hand detection, nail segmentation, and color application.

Optimize camera feed processing for real-time performance, especially on mobile devices.

Ensure that the app can handle real-time hand tracking and color application without performance degradation, especially for low-end devices.

Required Skills and Experience:

Strong experience in AR development (especially using MediaPipe and Flutter).

Expertise in 3D modeling and meshing (for creating realistic nail overlays).

Experience with machine learning and deep learning models, specifically for image segmentation or object detection.

Proficiency in Flutter and native mobile development (Android/iOS), with experience using platform channels to call native code.

Strong background in computer vision and experience with TensorFlow or PyTorch is a plus.

Familiarity with ARCore and ARKit for implementing AR features in Flutter.

Experience with camera integration and real-time video processing on mobile devices.

Ability to optimize performance for mobile applications, especially for AR and real-time processing.

Deliverables:

Working app with AR nail try-on functionality where users can select colors and apply them to the nails in real-time.

Fully functional nail segmentation (via MediaPipe or custom models).

Realistic 3D overlays on the nails using mesh generation and 3D curving techniques.

Optimized app with smooth real-time performance on mobile devices.

Documentation on the implementation, including setup instructions, API details, and usage guides.

We will be testing using xcode and we looking for the best price so dont just bid the higgest" (client-provided description)