Playrix

The impact of undervolting on Prompt Parsing and Token Generation speed on Blackwell 5090 RTX. Yesterday I've ran an experiment - a gradual decrease of power levels from 600w to 400w with a step of 25w on my 5090RTX. The inputs consisted of heavy prompts for mathematics, text analysis and deep thought, grouped in three identical blocks of 10k tokens each, totalling at 30k context size, similar to regular working conditions. Here are my findings: 1. KV-cache is retained between sessions on the same backend. A fresh session with the same input has been starting generation instantaneously, completely foregoing prompt parsing. This explains why running models in different processes improves quality of self-review drastically, while separate sessions can still miss a lot. 2. Warm cards lose 3-4% performance vs cold cards, therefore interactive mode will always feel a little faster than sustained load. 3. Max power (600w) vs min power (400w): Token Generation speed has decreased linearly, bottoming at -7%, while Prompt Parsing speed scaled non-linearly and has seen a change of -23%. 4. Max local inference load is still not using the 5090RTX fully, performance gains after 525w become asymptotic and negligent post 575w. The sweet spot of performance vs power therefore is at 525w. Grand total: For sustained loads where prompt parsing is heavily KV-cache reliant and does not load the GPUs the sweet spot is pretty much the minimum supported voltage. For interactive modes your mileage might vary, but going beyond 575w is practically pointless, while 525w might make your server room a little colder and quieter with little impact on your working mode. In one of the next posts I'll draw a comparison between 3090, 4090 and 5090, their power draw and price to performance. Let me know if you have 3090 or 4090 and will be willing to participate. In comments I'll link my post from Reddit with more graphs and the setup description. #llms #localinference #nvidia #genai #benchmark

There's a stark difference on the hiring market between the ML engineers who have effectively converted their experience into "years of launching AI-based features to production" and fellas who are just using AI as a tool. I'd say there are now three tiers of engineers who can still realistically find a job reasonably quick: Tier 1: ML engineers - working on revenue driving AI features, experience in production ready deployments. Top salaries, top demand, the big tech is actively hunting for you, the demand is skyrocketing due to almost any new startup now requiring at least a pair of such specialists. Tier 2: Platform and infra automation engineers - adopting AI to optimize cost drivers for the org. Good salaries, good demand, and a very dynamic environment now. Since you objectively need less of them per org and the amount of applicants is high, the salaries are lower than in Tier 1. The demand is growing slower than Tier 1, capped at the speed of the AI penetrating the late adopters market. Tier 3: Product engineers - using the AI to be more productive individually and compensate for the occasional lack of domain or tech knowledge. Salaries are stagnating, the demand is increasing at the expense of non-ai savvy roles. There will be a long tail of orgs converting to this mode of working for the years to come. Outside of these three categories the market has never been more difficult and having a steady job in this category will soon be seen as a luxury. #hiring #engineeringmanagement #genai #leadership #startups

Profiled local inference today on 5090RTX with different power levels. Will report soon, but let me tell you first that there were both some boring discoveries as well as some unexpected ones. Stay put :)

Hey folks, I have a few mentoring slots available. The first session is free. If you feel that you're overlooked for a promotion, stuck in a comfort zone or have a desire to switch tracks to leadership - let me know and we'll get you rolling. PS. I can issue official invoices, so we can make use of your learning budget =) #coaching #leadership #engineering #mentorship #personalgrowth #careerdevelopment