Rendered at 18:38:57 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
notatallshaw 5 hours ago [-]
As a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can.
But it has always had two major issues:
1. No way to reproduce exact distributions for a "download" command (there is no uv equivalent of "pip download")
2. For people with a lot of different environments the cache grows significantly more than pip
I'm interested to see, at least anecdotally, if this significantly improves 2, then we can perhaps have a two layer caching strategy without the significant disk space cost.
user5994461 4 hours ago [-]
Also as a pip contributor. The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv.
I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent.
It's very dangerous for empty files (init.py, empty.log yet not written). When the user edits one file, all files are modified simultaneously, all venv ever created by the user can be broken by editing one file, which is quite catastrophic.
It's also dangerous for small files with repeated content, for example random settings files that would contain a "1" or "true". Again, when the user edits one file, all files are edited and they were supposed to be different!
Hypothetically, a simple deduplication of binary files (.dll .so) should achieve 50% of the savings without significant drawbacks
I'd venture to say that pip extraction is more optimized than uv in at least one way. We have optimization for empty files (0 bytes) because there is nothing to write and checksum. uv doesn't seem to have the same optimizations, though I could be wrong, I just had a cursory look and my rust is not great. uv should probably review their treatment of empty files, it's counter productive to do any file system operation open/read/write because there is no content, it might be counterproductive to use any cache/comparison/hardlink if it takes more operations than doing nothing.
zanie 3 hours ago [-]
(I work on uv)
> The only advantage of uv is to have support for parallel async extraction.
This isn't true, the biggest speed ups are for the warm cases where we've already unpacked the files into the cache, as notatallshaw mentions above. We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code.
> If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv.
I'm a bit confused by these claims about the GIL? The expensive IO operations release the GIL.
> When the user edits one file, all files are modified simultaneously
This is why we default to reflinks or copy-on-write semantics when creating environments, not all file systems support it but it's becoming more common.
> Hypothetically, a simple deduplication of binary files (.dll .so) should achieve 50% of the savings without significant drawbacks
> We have optimization for empty files (0 bytes) because there is nothing to write and checksum.
Interesting, I would be very surprised if this made a significant difference? but I'll take a look.
notatallshaw 3 hours ago [-]
> We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code.
As a complete aside, uv can and does do this, but for this particular optimization I'm not sure how much absolute time it ends up saving compared to pure Python in real world resolution scenarios.
uv's total memory usage isn't that much leaner than pip's, and for parsing speed it turned out that the library pip uses, packaging, was just very unoptimized at the time uv launched. This has been significantly addressed since then:
At this point large dependency resolves in pip are spending very little of their time doing things in packaging, like version parsing. The main non-IO time spent in large resolves is now in the core resolver, resolvelib, which I hope to one day replace with my experimental resolver nab: https://github.com/notatallshaw/nab. Nab scales to large resolves much more efficiently than resolvelib (in fact I've cross-ported some of the algorithmic efficiency gains to uv already ;o)).
zanie 2 hours ago [-]
We didn't compare to pip at the time, but it saved a lot of absolute time for us as reported in the benchmarks from the pull request (https://github.com/astral-sh/uv/pull/789) it improved a boto3 case by 3x (30s to 10s) and our "standard" solve benchmark by 2x. It's plausible some of those gains have been reduced by other optimizations in our solver since then though.
But this was just one example optimization, we do other low-level things, like zero-copy deserialization from our cache. The point is not that we do specific things, but that we have more levers to pull to improve performance. It's great to see all the improvements happening in pip performance regardless :)
notatallshaw 2 hours ago [-]
> it improved a boto3 case by 3x (30s to 10s) and our "standard" solve benchmark by 2x. It's plausible some of those gains have been reduced by other optimizations in our solver since then though.
This makes a lot of sense because pubgrub makes heavy use of version comparison, compared to simple DFS algorithms like the one resolvelib uses. A lot of Pubgrub optimizations come from finding clever ways to not need to keep comparing versions.
> But this was just one example optimization, we do other low-level things, like zero-copy deserialization from our cache. The point is not that we do specific things, but that we have more levers to pull to improve performance.
Oh yes, I agree with the general point, I was just picking on the specific example for a fun exploration of performance optimizations.
Also, FWIW, I have in my professional career, not OSS work, implemented zero-copy deserialization from cache in pure Python, there are many surprising levers in Python when you are willing to explore the weird corners of the standard library.
user5994461 3 hours ago [-]
> I'm a bit confused by these claims about the GIL? The expensive IO operations release the GIL.
FYI: The extraction of files is largely python code that doesn't release the GIL. (cf. the zipfile class from the python interpreter has large layers of abstraction with a massive overhead in python code).
Regardless, python packages are thousands of tiny files, so pip never gets to release the GIL for any meaningful duration.
If you were writing an app that only extracted large GB files, you could take advantage of some I/O operations and some zlib operations freeing the GIL for a bit. Unfortunately pip is the opposite use case, lots of tiny files.
> Interesting, I would be very surprised if this made a significant difference? but I'll take a look.
Optimizing empty files was actually quite worthwhile for pip, because about 10% of python packages are empty init files.
This might not give the same result for uv though. pip is fully linear, every single open/read/write/stat operation we removed was a direct performance gain. uv does parallel async IO, you could very well remove 10% of filesystem calls and barely affect the overall duration. :D
tedivm 2 hours ago [-]
There are a ton of advantages to using uv over pip. I don't use UV because it's faster, although I do appreciate that. I use it because it's smart enough to manage virtual environments for each environment and tool, it can isolate to different python versions trivially, and it handles locking in a way that is actual sane.
jvolkman 2 hours ago [-]
> The only advantage of uv is to have support for parallel async extraction
Maybe the only advantage in a particular area (installing)? Because there are many other advantages.
The rich lock file for instance allows for much better cross-platform tooling. I can build a linux Docker container from a macos build host, for example, without VMs or any other emulation - simply using the cross-platform details in uv.lock and the correct tooling. I can even cross-compile numpy and other native wheels (linux -> macos, macos -> linux).
Other locker tools provide similar cross-platform information (Poetry, PDM), but pip is still lacking.
amelius 2 hours ago [-]
Imho deduplication belongs at the filesystem level, so the user won't (directly) see it or even know about it. Modern file systems like btrfs have the api for it.
scheme271 37 minutes ago [-]
The dedup functionality in something like zfs or btrfs isn't all that great. It tends to be extremely memory hungry and to slow things down significantly. E.g. ZFS needs around 1-5GB of ram per TB of storage and writes need to be compared to a hash table to dedup properly.
Using hints or knowledge at the app level is a much better experience if the app can tell the FS that two files are identical. The FS doesn't have to worry about hashing blocks within the file, correcting alignments, etc.
optionalsquid 3 hours ago [-]
> The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv.
That sounds like it would be relatively easy to demonstrate: You could tweak uv to perform downloads/extractions sequentially and then compare its runtime with pip. Has any such comparison been done?
zanie 3 hours ago [-]
I haven't done the comparison but it shouldn't be so hard
Good article. But the Makefile part should really be replaced by tox as a best practice. Tox even has a uv runner now that will handle setting up all the environments.
zbentley 5 hours ago [-]
There’s a bit of a bootstrap problem there in that Tox (and its dependencies) have to be installed first, unless your makefile is calling “uv run tox” or similar. Uv’s standalone nature makes it ideal for bootstrapping projects in a way that tox isn’t.
Zizizizz 4 hours ago [-]
I like just or mise tasks as often my python projects contain many other language commands (SQL, docker, pnpm, etc ...).
Timon3 34 minutes ago [-]
IMO mise tasks are fairly close to the ideal polyglot task runner, because features like argument definitions can be used as progressive enhancements. That means I can write the task logic in shell scripts that can be called without mise, and when I integrate them into mise, I don't have to manually write extra wrappers or duplicate anything. That's my pet peeve with most similar tools, so I can't recommend mise enough!
1aj-187 5 hours ago [-]
Can uv even be used with C extensions now?
zbentley 5 hours ago [-]
It always could.
tingletech 5 hours ago [-]
they just implement the modern PEPs
woodruffw 4 hours ago [-]
We do a lot more than that, e.g. how to cache distributions is not defined anywhere within PEPs (and should not be, since it’s a purely internal tool decision). But yes, it helps that uv implements the PEP for detached metadata, particularly during resolution.
310298 5 hours ago [-]
Nonsense. uv is one of many existing build tools. All the items can be done by other programs.
Funny to list OpenAI as an example at the end of the blog post ...
mark_l_watson 4 hours ago [-]
Nice improvement. For me, uv is the ‘Quicklisp for Python.’ uv just let me enjoy using Python like Quicklisp just makes Common Lisp nicer to use.
I have always been a Lisp devotee, but a few years ago when I started using uv, I then started seeing Python as a language I could really enjoy using so I put effort into making my Python dev setup nearly frictionless.
CivBase 7 hours ago [-]
A 10% reduction in cache size in exchange for a 4% slowdown doesn't seem obviously worthwhile to me, especially when it comes at an increase in complexity.
mminer237 5 hours ago [-]
Percentages aren't always the right gauge. uv isn't something I run frequently or for long periods of time. It's like a couple seconds every month or whatever. I would rather spend an extra second waiting every year to have hundreds more megabytes all the time.
Xirdus 5 hours ago [-]
When actively developing Python projects, running uv a dozen times per hour isn't unusual. Those seconds add up fast. Whereas freed disk space remains unused.
pas 3 hours ago [-]
warm cache case is only imperceptibly slower, no?
Xirdus 3 hours ago [-]
True, haven't read the source and only went by what the other commenter said. But one could argue 500MB more free space is similarly imperceptible.
kekebo 4 hours ago [-]
You could reverse the last sentence to "freed disk space reduces the need to clean the hd".
And I don't even mean physical limitation as much as the compulsion to manually clean ballooning cache dirs once in a while.
Xirdus 3 hours ago [-]
Overprovisioning also reduces the need to clean the disk. I've had a terabyte Macbook for over a year, developing in 7 different tech stacks simultaneously, never cleaning anything up manually, and I still have over 60% space left.
It's a tradeoff between wasting a little time everyday vs. wasting equally little time every year or so but also having the mental load of cleaning up plus maybe existential dread vs. spending more money to never have either problem. I do understand the last one isn't an option for everyone, but if it is, it's absolutely worth it.
charliermarsh 4 hours ago [-]
Making things faster is much easier than making things smaller. I'm sure we can win back a 4% slowdown elsewhere since we have so many more levers to pull from.
delduca 5 hours ago [-]
>especially when it comes at an increase in complexity.
Or (nasty) bugs hard to debug
colechristensen 6 hours ago [-]
Whereas I made a mistake and bought an underspeced MacBook and my 512GB disk is constantly on the brink of full as I reclaim the last 40 GB over and over from different caches, downloads, and wherever else and had to simply give up on several projects because of their disk usage.
Disk isn't free, especially now.
NoboruWataya 6 hours ago [-]
Exactly my experience, even worse because for me it's 2x256GB drives. Eventually I bit the bullet and upgraded one of them to 1TB, but not the one my root partition is mounted on (too lazy for that), so now I find myself moving various large directories across to my "extra" drive and symlinking them. Funnily enough I don't remember this ever being a problem back when 50GB was considered a lot.
drfloyd51 4 hours ago [-]
Because when 50gb was a lot, by the time you filled it, 200 GB was cheap and a non event.
db48x 5 hours ago [-]
Yea, I remember when the family computer had a 40MB hard drive and it was never ever filled to capacity. Now I’m wondering how many new disks to add to my ZFS pool, and when. And how to budget for it.
CivBase 6 hours ago [-]
I don't disagree. But if you're in that position, surely you'd be better off emptying the cache and disabling it?
colechristensen 6 hours ago [-]
The things which use my disk are things like package caches in several languages, build artifacts in both release and debug, multiplied by worktrees, things like iOS device support and release files + simulators, docker containers and related.
I had to give up on things like Lean theorem prover projects and VM projects because the disk space wasn't there.
The causes of disk waste are manifold and usually can't just be disabled and even when they can the result is constant rebuilding, redownloading, etc.
Too many developer products act like disk and memory is free to waste and the consequences have gotten ridiculous.
TacticalCoder 8 hours ago [-]
> deduplication at the file level: every file is now stored under its BLAKE3 hash
Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing).
If I've got a file named:
DSC98731-b3-7b39197a22.JPG
then:
- if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped me troubleshoot issues)
- if any other file has the same Blake3 7b39197a22 hash, it's a duplicate
- if that 7b39197a22 checksum is in my database, "things can happen".
For example my DB can say "any file with a Blake3 hash of 7b39197a22 can always be deleted" or "any file with a Blake3 hash of 887463c09e, if it's got a generic filename like "dscXXXXX" can always be renamed to "20260722jackJohnAtTheBeach-b3-778463c09e.jpg" (or whatever suits you).
It's really great (and I know several here independently made similar schemes) and Blake3 is an amazing hash for those kind of use.
Someone 8 hours ago [-]
For those wondering like me: Blake3 generates hashes of at least 224 bits, not, as a literal reading of that comment indicates, 40 bits (which would be bad for file deduplication, giving you a 50% hash collision after around a million files)
TacticalCoder 5 hours ago [-]
Yup sorry if I mislead people. I usually use 10 hexdigits (40 bits) but no matter how many there are, it's the x bits of the beginning of the checksum that are verified against the hash (in the examples I gave 40 bits).
I wasn't very clear.
pama 3 hours ago [-]
So at 3 million different files you have a 98.3% chance of a hash collision. Wouldnt that cause problems in real datasets?
maeln 7 hours ago [-]
Another cool thing about BLAKE3 is that it is a merkle tree. Not only it allows for good parallelism, but it also has a lot of cool property for data transfer. For example, you can check for partial validity, which allow for streaming error-detection and resend during the transfer. You only need to have the data and checksum in a way that you can start to reconstruct one or more subtree.
gchamonlive 6 hours ago [-]
> I use it for my own "deduplication / integrity / berzerker" utility
Do you have it in a public repo you could share?
TacticalCoder 5 hours ago [-]
No sadly I don't have any public repo: it's mostly really a collection of shell scripts and then the database one (for the berzerker/renamer) is in Clojure and I run it from my always-on REPL.
But I know others did similar thing so maybe there are public repos out there. But in any case: it should now be the kind of thing relatively easy to vibe-code if it's for your own use.
dist-epoch 8 hours ago [-]
It's annoying that most file formats don't checksum their own content.
Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.
PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18.
This is one great benefit of using .zip files as file formats, you get this for free.
OskarS 7 hours ago [-]
I think it's reasonable for a DB like SQLite to delegate that to the filesystem. There is an overhead for doing it on the DB level, and since SQLite is just a file on the filesystem which, presumably, is serving many other files as well, why would you trust anything else on the filesystem if you don't trust SQLite? Like, your PHP script (or nginx server executable, or whatever) that is calling SQLite, that's not going to be check-summed either. Either you trust your filesystem or you don't, and if you don't, checksum and error correct on the filesystem level.
Though fair enough, it could offer it as an opt-in thing.
coldtea 6 hours ago [-]
Because SQLite is a program, that will refuse to start or crash, and which you can trivially replace, if corrupted.
Whereas your sqlite data are your data, and if they're corrupted they can be lost forever or propagate the issue to backups.
conradludgate 2 hours ago [-]
Hopefully it crashes if corrupt... But it could instead jump straight to some code designed to clean up files and delete your database
SQLite 4 hours ago [-]
Checksums use CPU cycles. SQLite will do checksums with an extension (https://sqlite.org/cksumvfs.html) but that is off by default since an overwhelming majority of developers are more interested in day-to-day performance than detecting (very rare) storage malfunctions.
dist-epoch 3 hours ago [-]
Thank you for creating it!
I wonder if something changed, with todays SSDs, memory bottlenecked CPUs, and hardware accelerated CRCs.
FWIW, RAM bit-flips are much more common today, on consumer devices where SQLite is used a lot, since today's memory operates at the limit (see RowHammer).
globular-toast 39 minutes ago [-]
If you use a filesystem that does checksums, like ZFS, then you'd end up doing it twice for no reason. It makes more sense for archival formats like FLAC, though.
TacticalCoder 5 hours ago [-]
> It's annoying that most file formats don't checksum their own content.
I agree.
> Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.
I mostly run ext4 (desktop, laptops, etc.) but for my main server at home, it's a ZFS (mirrored) tank on an old server with ECC RAM.
And backups. So much backups.
dboreham 8 hours ago [-]
People say uv is good because it's fast, but honestly I don't care about that. We switched to using it for distribution of our Python-based tools because it makes it very convenient to install directly from a git repository and then to subsequently update from same repository. No need to build a package.
nchammas 6 hours ago [-]
Maybe I've misunderstood you, but that's not a special advantage of uv. pip has been able to do this for many years.
pip install git+https://github.com/some-org/repo
5 hours ago [-]
maeln 3 hours ago [-]
For me, the speed is the least interesting part of `uv`. What I like about it is what it bundle into a single bin : Managing installed python version, automatically creating local .venv for the project (I don't like software like pipenv who install the venv who knows where in a global directory of their choosing), support of pyproject.toml, including of the python version declared in it, and running script with dependencies.
None of it is unique to `uv` I believe, but it does it all, reliably, is easy to install, and easy to use. In terms of DX, for my use cases, it beat poetry, pipenv, pyenv (for python version management) and just using pip
BrokenCogs 1 hours ago [-]
For a uv noob like me: what do you mean by bundling it all into a single binary? How does that work?
MayeulC 18 minutes ago [-]
UV can be built (or downloaded) as a single optionally statically linked with the libc executable file, with no other dependencies. That makes it very easy to distribute on weird systems.
bulatb 30 minutes ago [-]
I think they mean that uv is a single tool which handles everything, instead of chaining pyenv, venv, pip-tools, pip, etc.
Neywiny 8 hours ago [-]
I tried it out successfully for the first time the other day (had 1 false start some months ago). This is after 10 ish years of system wide installs or venvs. I didn't find it fast at all. Every time I went to run the script it spent multiple seconds checking dependencies. Then one time it updated one, which luckily didn't break anything but I did get concerned. I'm sure there're some flags I didn't know to use but uvx was not great. On the other hand, it did seem to install the packages faster than pip.
nmstoker 8 hours ago [-]
YMMV depending on specific needs but generally even on Windows on an average machine with an imperfect setup I've found it exceptionally fast, often well inside the sub-second range, to the point that I do sometimes worry if it really ran.
Sounds like it's worth another look at your settings to make sure they are right.
coldtea 6 hours ago [-]
>I didn't find it fast at all.
Compared to what? Fast is relative. Compared to pip it's miles ahead.
Neywiny 3 hours ago [-]
Compared to just running the script with system-wide installs or in a venv, as I mentioned originally
intoXbox 7 hours ago [-]
I do avoid uv run for this reason but it’s useful for managing python projects. The speed claims actually have a lot to do with efficient caching, and I run several projects on the same Python (patch) version with similar packages on the same system
zanie 3 hours ago [-]
If you want to open an issue with verbose logs (`-vv`) I'd be happy to look into it. You can set `UV_LOG_CONTEXT=1` to emit timing information.
If you're running Python scripts, I'd recommend using `uv lock --script <path>` to generate a lockfile — we don't do that by default for scripts yet but that will avoid unexpected upgrades.
(I work on uv)
apple1417 5 hours ago [-]
I have never gotten the speed argument either. Yes it's noticeably faster - but in all my projects it's 1s vs 0.1s, not enough to make a meaningful difference. Even in their benchmarks, the worst number they ever give for pip is 7s [1] (and those numbers are 2+ years old, with python's improvements pip should be doing a little better now). I just can't say I've ever cared about installing dependencies, something you do once, taking 7s longer. Nor do I care how long CI takes, it running in the background is the point.
The one use I can give it is for running scripts with inline dependencies. I have found it a noticeable improvement over pipx there. But that's more due to needing to parse dependencies every single launch, if it's something I use regularly I end up just installing them normally instead, and it's faster than either.
This is exactly the kind of brief summary I was looking for to get my team more organized, thanks for this
as1297kj 7 hours ago [-]
What is the point of using packaging software that belongs to ClosedAI?
Astral people are already in the comments making pro AI coding statements. By using uv, you are literally supporting the people who want to make you unemployed for stock options.
embedding-shape 7 hours ago [-]
> By using uv, you are literally supporting the people who want to make you unemployed for stock options.
By using any FOSS or software produced by software developers and programmers, you are supporting the idea that computation can and should be automated, instead of having humans doing it. This is the origin of computing, and what we've been doing so far, and it continues to "eat the world" via automation, just like the past decades of it.
maeln 7 hours ago [-]
It's so human that many software engineers started to worry about automation and being replaced when it started to be something that might affect them.
ashg1-qwzudg 5 hours ago [-]
And it is machine like that many software engineers still cheer on being replaced once it is happening.
Software engineers are the socially dumbest creatures on the planet and deserve every bit of scorn that was heaped on them at high school. The industry used them for multi level marketing since about 2010 and now has instructed them to direct the MLM against themselves.
And they comply!
ashg1-qwzudg 5 hours ago [-]
That is a false equivalency. FOSS started as a counter movement against Microsoft in order to escape from surveillance and corporate control.
The AI cartel is the new Microsoft.
It is amazing how software developers have been brainwashed. Ted Ts'o, Google employee and pro-AI shill at Debian and LWN, recently argued that supporting AI is similar to supporting the Internet in 2000.
That is another fallacy. The Internet is a common carrier and was supposed to lead to freedom in 2000. It has nothing to do with outsourcing your thinking to a bunch of wannabe trillionaires.
embedding-shape 5 hours ago [-]
> That is a false equivalency. FOSS started as a counter movement against Microsoft in order to escape from surveillance and corporate control.
"Programming" as a concept started with humans being "tired" of manually calculating things, then it became faster than humans, and now our sand can talk human language.
And speaking about false equivalency, there are plenty of models you can run locally yourself. I agree that big tech sucks, and many places in the world need less surveillance and less corporate control, but I don't think the entire LLM-space (besides very obviously the biggest companies) are responsible for that kind of usage.
beepbooptheory 5 hours ago [-]
Is 'tired' here in scare quotes or are you referencing something? And if it wasn't faster from the get-go, how did it address the fatigue?
sharifhsn 4 hours ago [-]
The Internet relies on physical infrastructure managed by sovereign nations and multinational corporations, it would never be free from those constraints. Until we reach post-scarcity utopia and harmony between all of humanity—ironically, something that could be enabled by AI—the Internet will never be "free".
LeBit 7 hours ago [-]
What is the word? "Facetious"?
brnt 4 hours ago [-]
Maybe you left the L out on purpose, maybe you didn't. In any case: the L is about who is (or is able to) doing the automating. Users, not programmers, should be in charge.
ForceBru 6 hours ago [-]
Well, it's fast. Perhaps "they" want me and everyone else unemployed, but it's not immediate and not particularly certain. However, I can use uv here and now to greatly simplify my workflows, so I use it.
never_inline 5 hours ago [-]
If you're going to be unemployed anyway, and they don't gain a single penny by you using it, then what does this protest gain?
1aj-187 5 hours ago [-]
Wrong assumptions:
- "unemployed anyway" does not appear in the post you reply to.
- They gain market share and goodwill from fools. Not all gains have to be immediate monetary gains.
You could as well say that using ChatGPT is beneficial since it causes losses to OpenAI right now.
never_inline 3 hours ago [-]
It's laughable to suggest few people can change OpenAI's destiny by not using uv. Even if Astral goes bottom up, nothing happens OpenAI. So it's not like using ChatGPT.
rjzzleep 6 hours ago [-]
I think `pdm` is much nicer. I think if pdm had become a think earlier people would have switched from poetry to pdm rather than this uv thing which is a lot of different things at once.
zanie 3 hours ago [-]
I'm a bit confused. pdm hit 1.0 in 2021, 3 years before uv was released.
Chris2048 7 hours ago [-]
> Astral people are already in the comments making pro AI coding statements.
I can't see any other comment on this post mentioning AI, did you just make this up?
woodruffw 5 hours ago [-]
I think this sockpuppet is referring to a different thread I commented in yesterday. But I think it would be an uncharitable stretch to read what I wrote in an official voice, much less “pro” in an unqualified sense.
But it has always had two major issues:
1. No way to reproduce exact distributions for a "download" command (there is no uv equivalent of "pip download")
2. For people with a lot of different environments the cache grows significantly more than pip
I'm interested to see, at least anecdotally, if this significantly improves 2, then we can perhaps have a two layer caching strategy without the significant disk space cost.
I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent.
It's very dangerous for empty files (init.py, empty.log yet not written). When the user edits one file, all files are modified simultaneously, all venv ever created by the user can be broken by editing one file, which is quite catastrophic.
It's also dangerous for small files with repeated content, for example random settings files that would contain a "1" or "true". Again, when the user edits one file, all files are edited and they were supposed to be different!
Hypothetically, a simple deduplication of binary files (.dll .so) should achieve 50% of the savings without significant drawbacks
I'd venture to say that pip extraction is more optimized than uv in at least one way. We have optimization for empty files (0 bytes) because there is nothing to write and checksum. uv doesn't seem to have the same optimizations, though I could be wrong, I just had a cursory look and my rust is not great. uv should probably review their treatment of empty files, it's counter productive to do any file system operation open/read/write because there is no content, it might be counterproductive to use any cache/comparison/hardlink if it takes more operations than doing nothing.
> The only advantage of uv is to have support for parallel async extraction.
This isn't true, the biggest speed ups are for the warm cases where we've already unpacked the files into the cache, as notatallshaw mentions above. We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code.
> If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv.
I'm a bit confused by these claims about the GIL? The expensive IO operations release the GIL.
> When the user edits one file, all files are modified simultaneously
This is why we default to reflinks or copy-on-write semantics when creating environments, not all file systems support it but it's becoming more common.
> Hypothetically, a simple deduplication of binary files (.dll .so) should achieve 50% of the savings without significant drawbacks
We also explored this (see https://github.com/astral-sh/uv/pull/19694) and the linked pull request has a table comparing to this strategy.
> We have optimization for empty files (0 bytes) because there is nothing to write and checksum.
Interesting, I would be very surprised if this made a significant difference? but I'll take a look.
As a complete aside, uv can and does do this, but for this particular optimization I'm not sure how much absolute time it ends up saving compared to pure Python in real world resolution scenarios.
uv's total memory usage isn't that much leaner than pip's, and for parsing speed it turned out that the library pip uses, packaging, was just very unoptimized at the time uv launched. This has been significantly addressed since then:
* We did a lot of work to make version parsing twice as fast: https://iscinumpy.dev/post/packaging-faster/
* Since that blog post I made typical version parse three times faster on top of that: https://github.com/pypa/packaging/pull/1082
* Also since that blog post version filtering has gone through multiple optimizations and in some cases will be more than 30x faster e.g. https://github.com/pypa/packaging/pull/1105, https://github.com/pypa/packaging/pull/1111, https://github.com/pypa/packaging/pull/1120
At this point large dependency resolves in pip are spending very little of their time doing things in packaging, like version parsing. The main non-IO time spent in large resolves is now in the core resolver, resolvelib, which I hope to one day replace with my experimental resolver nab: https://github.com/notatallshaw/nab. Nab scales to large resolves much more efficiently than resolvelib (in fact I've cross-ported some of the algorithmic efficiency gains to uv already ;o)).
But this was just one example optimization, we do other low-level things, like zero-copy deserialization from our cache. The point is not that we do specific things, but that we have more levers to pull to improve performance. It's great to see all the improvements happening in pip performance regardless :)
This makes a lot of sense because pubgrub makes heavy use of version comparison, compared to simple DFS algorithms like the one resolvelib uses. A lot of Pubgrub optimizations come from finding clever ways to not need to keep comparing versions.
> But this was just one example optimization, we do other low-level things, like zero-copy deserialization from our cache. The point is not that we do specific things, but that we have more levers to pull to improve performance.
Oh yes, I agree with the general point, I was just picking on the specific example for a fun exploration of performance optimizations.
Also, FWIW, I have in my professional career, not OSS work, implemented zero-copy deserialization from cache in pure Python, there are many surprising levers in Python when you are willing to explore the weird corners of the standard library.
FYI: The extraction of files is largely python code that doesn't release the GIL. (cf. the zipfile class from the python interpreter has large layers of abstraction with a massive overhead in python code).
Regardless, python packages are thousands of tiny files, so pip never gets to release the GIL for any meaningful duration.
If you were writing an app that only extracted large GB files, you could take advantage of some I/O operations and some zlib operations freeing the GIL for a bit. Unfortunately pip is the opposite use case, lots of tiny files.
> Interesting, I would be very surprised if this made a significant difference? but I'll take a look.
Optimizing empty files was actually quite worthwhile for pip, because about 10% of python packages are empty init files.
This might not give the same result for uv though. pip is fully linear, every single open/read/write/stat operation we removed was a direct performance gain. uv does parallel async IO, you could very well remove 10% of filesystem calls and barely affect the overall duration. :D
Maybe the only advantage in a particular area (installing)? Because there are many other advantages.
The rich lock file for instance allows for much better cross-platform tooling. I can build a linux Docker container from a macos build host, for example, without VMs or any other emulation - simply using the cross-platform details in uv.lock and the correct tooling. I can even cross-compile numpy and other native wheels (linux -> macos, macos -> linux).
Other locker tools provide similar cross-platform information (Poetry, PDM), but pip is still lacking.
Using hints or knowledge at the app level is a much better experience if the app can tell the FS that two files are identical. The FS doesn't have to worry about hashing blocks within the file, correcting alignments, etc.
That sounds like it would be relatively easy to demonstrate: You could tweak uv to perform downloads/extractions sequentially and then compare its runtime with pip. Has any such comparison been done?
- https://docs.astral.sh/uv/reference/environment/#uv_concurre...
- https://docs.astral.sh/uv/reference/environment/#uv_concurre...
This one being downloading to an offline wheelhouse and installing from that.
The other one being having a shared named global environment ;o).
P.S. I'll have to remove this as an important feature nab has that uv doesn't when I make the announcement nab is no longer experimental, aha.
https://stephenlf.dev/blog/python-library-in-2026/
Funny to list OpenAI as an example at the end of the blog post ...
I have always been a Lisp devotee, but a few years ago when I started using uv, I then started seeing Python as a language I could really enjoy using so I put effort into making my Python dev setup nearly frictionless.
It's a tradeoff between wasting a little time everyday vs. wasting equally little time every year or so but also having the mental load of cleaning up plus maybe existential dread vs. spending more money to never have either problem. I do understand the last one isn't an option for everyone, but if it is, it's absolutely worth it.
Or (nasty) bugs hard to debug
Disk isn't free, especially now.
I had to give up on things like Lean theorem prover projects and VM projects because the disk space wasn't there.
The causes of disk waste are manifold and usually can't just be disabled and even when they can the result is constant rebuilding, redownloading, etc.
Too many developer products act like disk and memory is free to waste and the consequences have gotten ridiculous.
Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing).
If I've got a file named:
then: For example my DB can say "any file with a Blake3 hash of 7b39197a22 can always be deleted" or "any file with a Blake3 hash of 887463c09e, if it's got a generic filename like "dscXXXXX" can always be renamed to "20260722jackJohnAtTheBeach-b3-778463c09e.jpg" (or whatever suits you).It's really great (and I know several here independently made similar schemes) and Blake3 is an amazing hash for those kind of use.
I wasn't very clear.
Do you have it in a public repo you could share?
But I know others did similar thing so maybe there are public repos out there. But in any case: it should now be the kind of thing relatively easy to vibe-code if it's for your own use.
Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.
PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18.
This is one great benefit of using .zip files as file formats, you get this for free.
Though fair enough, it could offer it as an opt-in thing.
Whereas your sqlite data are your data, and if they're corrupted they can be lost forever or propagate the issue to backups.
I wonder if something changed, with todays SSDs, memory bottlenecked CPUs, and hardware accelerated CRCs.
FWIW, RAM bit-flips are much more common today, on consumer devices where SQLite is used a lot, since today's memory operates at the limit (see RowHammer).
I agree.
> Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.
I mostly run ext4 (desktop, laptops, etc.) but for my main server at home, it's a ZFS (mirrored) tank on an old server with ECC RAM.
And backups. So much backups.
None of it is unique to `uv` I believe, but it does it all, reliably, is easy to install, and easy to use. In terms of DX, for my use cases, it beat poetry, pipenv, pyenv (for python version management) and just using pip
Sounds like it's worth another look at your settings to make sure they are right.
Compared to what? Fast is relative. Compared to pip it's miles ahead.
If you're running Python scripts, I'd recommend using `uv lock --script <path>` to generate a lockfile — we don't do that by default for scripts yet but that will avoid unexpected upgrades.
(I work on uv)
The one use I can give it is for running scripts with inline dependencies. I have found it a noticeable improvement over pipx there. But that's more due to needing to parse dependencies every single launch, if it's something I use regularly I end up just installing them normally instead, and it's faster than either.
[1] https://github.com/astral-sh/uv/blob/main/BENCHMARKS.md
https://stephenlf.dev/blog/python-library-in-2026/
Astral people are already in the comments making pro AI coding statements. By using uv, you are literally supporting the people who want to make you unemployed for stock options.
By using any FOSS or software produced by software developers and programmers, you are supporting the idea that computation can and should be automated, instead of having humans doing it. This is the origin of computing, and what we've been doing so far, and it continues to "eat the world" via automation, just like the past decades of it.
Software engineers are the socially dumbest creatures on the planet and deserve every bit of scorn that was heaped on them at high school. The industry used them for multi level marketing since about 2010 and now has instructed them to direct the MLM against themselves.
And they comply!
The AI cartel is the new Microsoft.
It is amazing how software developers have been brainwashed. Ted Ts'o, Google employee and pro-AI shill at Debian and LWN, recently argued that supporting AI is similar to supporting the Internet in 2000.
That is another fallacy. The Internet is a common carrier and was supposed to lead to freedom in 2000. It has nothing to do with outsourcing your thinking to a bunch of wannabe trillionaires.
"Programming" as a concept started with humans being "tired" of manually calculating things, then it became faster than humans, and now our sand can talk human language.
And speaking about false equivalency, there are plenty of models you can run locally yourself. I agree that big tech sucks, and many places in the world need less surveillance and less corporate control, but I don't think the entire LLM-space (besides very obviously the biggest companies) are responsible for that kind of usage.
- "unemployed anyway" does not appear in the post you reply to.
- They gain market share and goodwill from fools. Not all gains have to be immediate monetary gains.
You could as well say that using ChatGPT is beneficial since it causes losses to OpenAI right now.
I can't see any other comment on this post mentioning AI, did you just make this up?