I don't really know what kind of rebuttal you're looking for, but I will link my HN comments from when this was first posted for some thoughts: https://news.ycombinator.com/item?id=31396861#31398796. As I said, in the linked post, I'm quite skeptical of the business of trying to assess relative buginess of programming in different systems, because that has strong dependencies on what you consider core vs packages and what exactly you're trying to do.
However, bugs in general suck and we've been thinking a fair bit about what additional tooling the language could provide to help people avoid the classes of bugs that Yuri encountered in the post.
The biggest class of problems in the blog post, is that it's pretty clear that `@inbounds` (and I will extend this to `@assume_effects`, even though that wasn't around when Yuri wrote his post) is problematic, because it's too hard (arguably impossible) to write correctly. My proposal for what to do instead is at https://github.com/JuliaLang/julia/pull/50641.
Another common theme is that while Julia is great at composition, it's not clear what's expected to work and what isn't, because the interfaces are informal and not checked. This is a hard design problem, because it's quite close to the reasons why Julia works well. My current thoughts on that are here: https://github.com/Keno/InterfaceSpecs.jl but there's other proposals also.
> Another common theme is that while Julia is great at composition, it's not clear what's expected to work and what isn't, because the interfaces are informal and not checked.
This is THE huge issue when combined with the other variables:
- you often have hundreds of depedencies that interlock
- you often have dependencies that work with multiple packages, and they will often pull them all into your tree (there are sometimes "bridge packages" written, but that is O(n^2) in the number of packages)
- many maintainers do not care about good versioning practices, because they are not enforced, and it is not their problem when they break a bajillion other packages
- many people writing Julia code are not software developers. this is great! but they usually don't look at what their testing coverage looks like. often the CI always fails on a package and it's ignored.
Would Julia only allow multiple dispatch as a link between those packages, the situation would look a little better. But often the packages also talk through simple accesses of values like `X.someproperty`. I've seen situations where maintainers would add and remove these properties and break their own other libraries. Better "enforcement" of these types of things - however that would look like - would be a huge improvement for the time sink that is maintaining a Julia package.
> But often the packages also talk through simple accesses of values like `X.someproperty`. I've seen situations where maintainers would add and remove these properties and break their own other libraries. Better "enforcement" of these types of things - however that would look like - would be a huge improvement for the time sink that is maintaining a Julia package.
I think this is a cultural issue to a large degree. I think even if there were better enforcement, it's just a question of coding of discipline to then actually not break something. If it were easier to anticipate what sort of breakage a change could entail (say, by making clear & documented _by the core language_ what is considered a breaking change, and then actually not even doing "technically breaking" changes and using deprecations instead), this COULD change.
That being said, that requires a very different development workflow than what is currently practiced in the main repos of the language, so there's quite a bit of an uphill battle to be had (though such issues happen there all the time, so solving that would actually help here the most, ironically).
That sounds like Julia needs some kind of test suite, that lives alongside the major libraries used in Julia, and verifies that things that interact do so in a reasonable manner e.g. things that implement addition obey the rules of addition, etc, etc.
With a little reflection and some automation, such a tool could be taught to test new Julia libraries and catch problems early.
i.e. what I am talking about is effectively informally encoding protocols/interfaces and suchlike in a test-suite, rather than seeking to make it a part of the compiler/language.
Not as sexy, no doubt, but flexible and with a short time to get feedback.
This is precisely what we did with the SciML ecosystem. You can see a blog post from a year back: https://sciml.ai/news/2022/10/08/error_messages/. There's a lot of high level interface checking that goes on now through a trait system to ensure that the pieces are solvable before hitting code, and throwing high level error messages back based on those.
Over the last year we've been growing the domain of these as well, and have seen a decrease in bug reports come from it.
There is no rebuttal because nothing much has really changed culture wise. Sure, the various @inbounds issues and concrete bugs that are mentioned in Yuris post have mostly been addressed, but the larger point (that is, "what can I actually expect/get guaranteed when calling a given function?") definitely hasn't been, at least not culturally (there are parts of the ecosystem that are better at this, of course). Documentation of pre- and postconditions are still lackluster, PRs trying to establish that for functions in Base stall for unclear reasons/don't get followups and when you try to talk about that on Slack retorts boil down to "we're tired of hearing you complain about this" instead of trying to find a systemic solution to that problem. Until that changes, I have large doubts about Yuris post losing relevance.
My own efforts (shameless plug, https://github.com/Seelengrab/PropCheck.jl for property based testing inspired by Hedgehog and https://github.com/Seelengrab/RequiredInterfaces.jl for somewhat formalizing "what methods are needed to subtype an abstract type") are unused in the wider community as far as I can tell, in spite of people speaking highly of them when coming across them. I also don't think Kenos InterfaceSpecs.jl is the way forward either - I think there's quite a lot of design space left in the typesystem the language could do without reaching for z3 and other SAT/SMT solvers. I personally attribute the lack of progress on that front to the lack of coherent direction of the project at large (and specifically not to the failings of individuals - folks are always very busy with their lives outside of Julia development/other priorities). In spite of the fact that making this single area better could be a big boon with more traditional software engineers, which are very underrepresented in the community.
There is a culture of well-defined interfaces which are checked at compile time. This is something that was emphasized in recent posts and changes such as:
* SciMLStyle (https://github.com/SciML/SciMLStyle) came into existence and defines a style which avoids any of the behaviors seen in the blog post.
* Julia came out with package extensions in v1.9 (https://www.youtube.com/watch?v=TiIZlQhFzyk) and with the interface checking, implicit interface support was turned mostly into explicit interface support where packages and types opt-in via defining traits (this is still continuing to evolve but is mostly there).
Given all of these changes, most of the things in the blog post would now error in many standard packages without someone explicitly defining interface trait functions to allow an object in that doesn't satisfy the interface which it's claiming to. Of course, not every person or every package has changed, but what I described here are major interface, style, and cultural changes to some of the most widely used packages since 2020.
> with the interface checking, implicit interface support was turned mostly into explicit interface support where packages and types opt-in via defining traits (this is still continuing to evolve but is mostly there)
What interface checking? Base doesn't provide any such facilities. Package extensions are still ad-hoc constructions on a package by package basis; there is little to no consistency.
> Of course, not every person or every package has changed, but what I described here are major interface, style, and cultural changes to some of the most widely used packages since 2020.
And none of that has landed in Base, none of that is easy to find out about/discoverable for non-SciML users. SciML is not all of Julia, and certainly not Base or packages outside of SciML. Please don't frame this as if SciML was the only thing that mattered here.
Yes it has not landed in Base, but you're acting like there has not been a general cultural changes. I showed, with receipts, that many of the most widely used packages in the Julia ecosystem have adopted new infrastructure, systems, and tooling to address these problems in the last 3 years. With SciML and JuMP both having adopted such systems, this accounts for roughly 50% of the top 100 most starred Julia packages according to current metrics (Nov 20 2023), with many of the packages not doing this largely being interface packages (plotting, notebooks, and language interop, if you account for those not having this issue it's closer to 2/3 of the top 100 most starred packages). I also want that number to be 100%, and the compiler team is having weekly discussions with us about our needs given that there are now successful parts of the ecosystem to model this tooling based off of, and so yes it can get better and we need to keep improving. But to claim that no shift in culture has occurred is implying that all of this doesn't exist, even though we can point to the receipts.
50% of the top 100 most starred, i.e. used, packages are not representative of the entire community. Not to mention that the vast majority of those SciML packages is developed by a relatively small group of people, compared to the rest of the ecosystem. If all a potential user cares about is SciML, good for them! I've repeatedly said that users not looking for SciML are left behind.
Yes, SciML is doing good. I'm not denying that, and never have. Still, the rest of the community/package ecosystem is not good at catching up - which is what I'm criticizing.
This article is a litany github issues for correctness bugs in library code, every one of which appears to have been fixed long ago now so it's not quite true to say there's been no rebuttal. Whether there remain a lot of correctness bugs in julia library code I can't say, but not these ones any more.
Maybe sometime from the realm of psychology would like to theorize why this article became so popular? I never found it compelling.
I think it's because evaluating a language is horribly complicated, and people crave certainty and the article sounds very definive. Maybe people see it as a sort of the article is right/Julia bas vs the article is wrong/Julia good.
Besides the 'correctness' issues, all of the issues that blogpost mentioned were fixed back then ASAP. Most of it arise from the usage of @inbounds, which to be fair, the documentation clearly tells you its usage is unsafe. Is it bad there were unrealized bugs in the official example? Yes. Did it get fixed immediately once reported? Yes. Mistakes happen everywhere else. I don't like the stance the blogpost took at all, and I find it quite misleading as it falsely implies Julia is bug-ridden. Personally, the number of bugs I've faced in Python is far more than the ones I've run into in Julia.
It's not the fact that the bugs were fixed. It would be really strange to revisit this article and find out that they weren't, right? It's the fact that the author has spent years working with the language and published several libraries on the way. His article contains other links to similar experiences other ppl had with the language. There is another informative JAX vs Julia article there (by Patrick Kidger) as well. It's just enough material for all kinds of doubt and this really sucks for the language. I simply cannot image a situation where Julia will be seriously considered in a commercial company after this, not in prod. And what's worse, anyone who will fancy the idea of investing into a new language will have a hard time justifying its worth. After all, when you read about several months of hunting for bugs, you definitely don't want to be that person.
>And what's worse, anyone who will fancy the idea of investing into a new language will have a hard time justifying its worth. After all, when you read about several months of hunting for bugs, you definitely don't want to be that person.
Yea, that would be me. Julia seemed like a good fit but I don't want to spend months bug hunting someone else's language
I remember reading this article and being curious I looked at all the correctness issues. I think I found almost all of the issues have been resolved. Of course, they might've gotten priority after the publication of this article and might not mean that the fundamental issues are not their. But I think it is a sign that the language is in continuous improvement.
Regarding the @inbounds problem. If removing @inbounds leads to out of bounds memory access, doesn't that mean it was masking a bug? Shouldn't it crash with a message like an assert?
If @inbounds wasn't used, then yes, it would throw an error, but @inbounds is specifically for removing those error paths.
What happened here is that julia has a package called OffsetArrays.jl which can be used to create arrays with arbitrary index offsets. Julia arrays start at index 1 (1-based indexing), but with OffsetArrays you could make 0-based arrays, or 10-based arrays or whatever you want.
There happened to be a bunch of code laying around which was older than OffsetArrays.jl and did things like
function f(v::AbstractVector)
for i in 1:length(v)
@inbounds do_something(v[i])
end
end
which implictly assumed that all abstract vectors could be indexed according to normal 1-based indexing rules. This created a lot of silent problems when people then went and fed offset arrays into functions that were using such incorrect indexing schemes.
Instead, modern julia code that works on AbstractArray is supposed to be written using functions like eachindex, or axes, i.e.
function f(v::AbstractVector)
for i in eachindex(v)
@inbounds do_something(v[i])
end
end
though Julia's compiler is now smart enough that using @inbounds like this is typically unnecessary, since it can see that eachindex(v) can't produce an out-of-bounds index.
Couldn't this kind of things be caught by type checking?
1-indexed vectors should be different type than arbitrary offset vector (implicitly convertible in one way only).
They are different types, but they share a common supertype (AbstractArray). AbstractArray includes thing like regular Array, and also a whole zoo of other array types (including OffsetArray).
The problematic code was incorrectly saying it worked on any subtype of AbstractArray, but was actually written assuming that the arrays were 1-based which is an invalid assumption.
What do you mean precisely? Removing @inbounds should never leads to OOB access. The @inbounds macro disables the boundscheck.
If what you mean was that: If the act of removing an @inbounds leads to a bounds exception, then the code was buggy even without the macro. That's true. IIUC, the main issue that Yuri talks about is that Julia is a highly generic language where people share abstractions quite a lot, but there is little agreement on what those abstractions are, and the language provide zero tooling or help with that
>> What do you mean precisely? Removing @inbounds should never leads to OOB access. The @inbounds macro disables the boundscheck.
Oh, that explains it. I thought it enabled a bounds check that somehow allowed code to run rather than fault. So it is a promise to the compiler that the code is correct and doesn't need checking. Composition with arbitrary bounds silently breaks that promise. Go it.
I think the implication is that @inbounds turns out of bound memory accesses into undefined, silent behavior, which can stay hidden and cause weird bugs later on. Out of bounds without using @inbounds leads to an exception indeed.
I think those points are just the symptoms. The real problem is having an out of tree dependency on LLVM. It is too much work to keep up with. The LLVM project does not care about breaking their external users' code.
No, that has nothing to do with any of the problems julia faces. Julia's LLVM is just regular LLVM with some patches that are usually upstreamed. The Julia team is pretty good at keeping up with upstream LLVM.
It is the social problem of having people show up and grind maintenance work to just make it compile. Which means other questions are left untouched. It is approximately zero sum.
I mean, fundamentally, keeping a C++ abi backwards compatible is a horror story. It like, just doesn't work, for non-trivial projects. So LLVM breaks all the time.
LLVMs API is the c++ one. The C one while more stable also doesn't support everything. Keeping up with LLVM is annoying but it's not the source of bugs or anything of the sort. PS(it's not actually stable. Because if the c++ code it calls is removed it just gets removed from the C one)
I say this as one of the devs that usually do the work of keeping up the latest LLVM.
