Skip to content
Hallway
0:00 / 0:00

#athena#jpmorgan#monorepo

Python at Massive Scale - Stephen Simmons, Neil Slinger

PyData

173K subscribers • 11 videos on Hallway

YouTube

Intermediate. Assumes working knowledge.

9.9K views8 years ago1 recommendation

Programming languagesTools and practicePythonSoftware architectureCode review and teamwork

Stephen Simmons and Neil Slinger describe how JPMorgan scaled its Athena Python platform to 35 million lines of code and thousands of developers.

PyData London 2018 The talk describes how JPMorgan has scaled its Athena Python trading and risk analytics platform over 10 years to 35m LOC/150k Python modules, and several thousand developers. The talk will also cover key parts of the infrastructure and tooling for data analytics and visualization, and explain how those have been adapted for workloads orders of magnitude larger than originally envisaged. --- www.pydata.org PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R. PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases. Welcome! Help us add time stamps or captions to this video! See the description for details. Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: github.com/numfocus/YouTubeVideoTimestamps

Why it's here

1 person recommends this

What people say. Recommended as a rare look into managing a massive Python monorepo and developer workflow at a major financial institution.

stevesimmonson Hacker News • Recommends

On “The most remarkable legacy system I have seen”

As the comment by timsworkaccount says, this is definitely Athena at JPMorgan. For those interested, I gave a talk on Athena at PyData UK 2018 called "Python at Massive Scale". 4500 developers making 20k commits a week. Codebase with 35m LOC. The video is here: link to this video It covers Athena's origins, what it is used for, application architecture, infrastructure, dev tooling and culture.
View on Hacker News

stevesimmonson Hacker News • Mentions

On “Why Google Stores Billions of Lines of Code in a Single Repository (2016)”

My company has a 50m LOC Python codebase in a monorepo. It works really well, given the rate of change of thousands of developers globally. That is only possible because of the significant investment in devtools, testing and the deployment infrastructure. Here is "Python at Massive Scale", my talk about it at PyData London earlier this year: link to this video
View on Hacker News

mcintyre1994on Hacker News • Mentions

On “Python is becoming the world’s most popular programming language”

If anyone else is looking: link to this video :) I'm definitely interested in this - I've seen a struggle to manage a far smaller codebase than that in Python and really interested to see what you're doing differently to make it work.
View on Hacker News

Barrin92on Hacker News • Mentions

On “Python Is Eating the World”

>How stable is python to run a full trading / quantitative algorithm on? JP Morgan operates a ~30 million LOC Python platform for trading and analytics. (related talk: link to this video ) Yes, there are very, very large working python codebases in fields out there that demand correctness. I'm honestly getting tired of the static typing circlejerk that has entered the industry.
View on Hacker News

stevesimmonson Hacker News • Mentions

On “An oral history of Bank Python”

Nice stuff! For a view of Python at JPMorgan, I did a talk at PyData London "Python at Massive Scale" - link to this video
View on Hacker News

stevesimmonson Hacker News • Mentions

On “An oral history of Bank Python”

I worked on Athena at JPMorgan for 8 years, and loved it. Seeing Python at the core of trading, risk and post-trade processing for Commodities, FX, Credit etc was such a great developer experience. By the time I left JPM, there were 4500 devs globally making 20k commits weekly into the Athena codebase. (I did a PyData presentation on this 1 for more details). The one downside was the delayed transition from Py2.7 to 3; I left just as that was getting underway. 1 link to this video The
View on Hacker News

stevesimmonson Hacker News • Mentions

On “Problems once solved by a metaclass can be solved by __init_subclass__”

See 1 for a PyData London talk I did on "Python at Massive Scale", about JPMorgan's Athena trading and risk management system, used for FX, Commodities, Credit, Equities, etc trading. At that time, Athena had 4,500 Python developers doing 20,000 commits a week. Also see one recent HN discussion on "An Oral History of Bank Python" 2. 1 link to this video 2 link
View on Hacker News

Barrin92on Hacker News • Mentions

On “Modern Python Performance Considerations”

>Python shines beautifully as a (somewhat) cross platform scripting language Python is much more than just a scripting language. I remember attending this talk1 a few years about JPMorgan's 35 million LOC Python codebase. Python is being used to built seriously large software nowadays and I don't think performance is ever a minor issue. It should always be in the top 3 for any general purpose language because it directly translates into development speed, time and money. 1 link to this video
View on Hacker News