Senior Software Architect - Python/PySpark

Project Description :

We are looking for a Senior Architect to lead the technical direction of our Trade Surveillance platform. Partnering closely with Enterprise Architects, Compliance, and Quantitative teams, this role will shape and execute the multi-year trade surveillance technology roadmap covering scenario and alert development, platform modernization, and migration from a Q/KDB-based stack to a scalable Python/PySpark cloud-native architecture.

This is a direct and permanent role where the ideal candidate combines deep hands-on engineering credibility with the architectural maturity to influence stakeholders across business, compliance, and technology.

Skills required :

- 10+ years of technology experience in Investment Banking and/or Capital Markets sector

- Deep proficiency in Python or PySpark, including distributed processing patterns, performance tuning, and production-grade engineering.

- Hands-on experience with at least one major cloud platform (AWS, Azure, or GCP) and modern data platforms (Databricks, Snowflake, EMR, or equivalent).

- Strong grasp of streaming and event-driven architectures (Kafka, Flink, Kinesis, or similar).

- Understanding of trade lifecycle, market microstructure, and at least one regulatory regime (MAR, Dodd-Frank, MiFID II, SEBI, FINRA rules).

- Proven ability to operate at the intersection of engineering and architecture designing on the whiteboard and coding when needed.

- Experience with Generative AI / LLM frameworks (LangChain, LlamaIndex, vector databases, RAG patterns) and ML libraries (scikit-learn, PyTorch).

Key Competencies :

- Architectural thinking balanced with engineering pragmatism.

- Strong written and verbal communication; able to translate regulatory and business intent into technical design.

- Comfortable navigating ambiguity in a high-stakes, regulator-facing environment.

- Influences without authority credible with senior engineers, enterprise architects, and compliance leadership alike.

Responsibilities :

Architecture & Roadmap :

Partner with Enterprise Architects to define and evolve the trade surveillance technical roadmap, ensuring alignment with firm-wide architecture standards, data strategy, and regulatory expectations.

Own end-to-end architecture for surveillance scenario development, alert generation, case management integration, and downstream investigator tooling.

Define reference architectures, design patterns, and engineering guardrails for the surveillance engineering organization.

Scenario & Alert Engineering :

Lead the design and implementation of trade surveillance scenarios and alerts across asset classes (equities, fixed income, FX, derivatives) covering market abuse typologies such as spoofing, layering, wash trades, front-running, insider trading, and cross-product manipulation.

Drive development of scenarios and analytics in both Q/KDB (current state) and Python/PySpark (target state).

Establish reusable frameworks for scenario authoring, parameter tuning, backtesting, threshold calibration, and false-positive reduction.

Platform Modernization & Cloud Migration :

- Lead migration of the trade surveillance platform to the new cloud infrastructure (AWS / Azure / GCP), including compute, storage, streaming, and orchestration layers.

- Architect and oversee the migration of scenarios, libraries, and frameworks from Q/KDB to Python/PySpark on distributed compute (Spark, Databricks, EMR, or equivalent), ensuring functional parity, performance, and auditability.

- Design for scale alert generation across billions of order and trade events per day with focus on throughput, latency, cost optimization, and resilience.

Data & Engineering Excellence :

- Define data models and ingestion patterns for orders, executions, market data, reference data, communications, and news/social feeds.

- Champion engineering best practices: CI/CD, IaC, automated testing of surveillance logic, observability, lineage, and reproducibility of alerts (critical for regulatory defensibility).

- Collaborate with Data Engineering, DevOps, and InfoSec on secure-by-design implementations.

Innovation & GenAI :

- Identify and prototype applications of Generative AI and ML in surveillance narrative generation for alerts, investigator copilots, anomaly detection, communications surveillance (NLP), and intelligent triage.

- Evaluate vendor and open-source capabilities and guide build-vs-buy decisions.

Leadership & Stakeholder Management :

- Mentor senior engineers and tech leads; conduct design reviews and uphold architectural quality.

- Engage with Compliance, Front Office Supervision, Internal Audit, and Regulators on technical aspects of the surveillance program.

(ref:hirist.tech)