Building an AI-Powered HRMS: Architecture Lessons from NexHR
Mohsin Malik · Co-Founder & Data Architecture LeadNexHR is an HR management system that runs four distinct AI/ML subsystems, semantic resume screening, biometric face-recognition attendance, a zero-manual-step payroll engine, and a RAG-based HR assistant, inside a single product with five RBAC roles. The core architecture lesson is to treat each AI capability as an independently testable service with a deterministic fallback, and let boring infrastructure like PostgreSQL, Django, and Celery carry the integration load.
What did we actually build?
- Semantic resume screening, using BAAI/bge-base-en-v1.5 embeddings (768 dimensions) with cosine-similarity ranking of candidates against job descriptions.
- Biometric attendance, an OpenCV plus ONNX Runtime face-recognition pipeline with geofencing and a manual fallback path.
- Payroll engine, service aggregation over attendance, loans, and expenses, gross-to-net tax brackets, payslip snapshots, and Stripe Connect disbursements, with zero manual calculation steps.
- RAG HR assistant, company policy documents indexed in pgvector, answered by Google Gemini 2.0 Flash with grounded retrieval.
- Platform: Django, Celery, and PostgreSQL, a React frontend, JWT auth, and five RBAC roles (Admin, HR Manager, Finance Manager, Employee, Candidate).
How do you score resumes without hand-written rules?
Keyword matching fails on resumes because people describe the same skill twenty different ways. Embeddings fix this. We embed the job description and each candidate profile, then rank by a weighted score. Final Score = (0.50 × semantic similarity) + (0.30 × skills coverage) + (0.20 × experience match). The weights are deliberately visible and tunable rather than buried in a model, so HR can understand and defend a ranking. That explainability turned out to matter as much as accuracy.
Why run face recognition through ONNX?
Attendance check-in happens dozens of times a minute at peak, so inference cost and latency dominate. Exporting the recognition model to ONNX Runtime cut inference overhead compared to running a full framework, and pairing it with geofencing plus cosine-distance matching kept false accepts manageable. The equally important decision was the manual fallback. When recognition confidence is low, the flow drops to a supervised manual check-in instead of blocking a real employee at the door. Every AI feature in the product has a non-AI escape hatch to match it.
What makes a payroll engine "zero manual steps"?
Payroll is where AI products usually cheat, with a slick dashboard sitting over a spreadsheet process underneath. We built payroll as a service-aggregation pipeline. Attendance records, approved loans, and expense claims flow in. Gross-to-net tax bracket logic runs deterministically. A payslip snapshot gets frozen for audit. Stripe Connect handles disbursement. Celery orchestrates the runs asynchronously so a long payroll cycle never blocks the app. Determinism was non-negotiable here. Money math is the one place an LLM doesn't belong.