I gave an Erlang talk many years ago where I used the Tandem architecture as a comparison point, and one of the attendees happened to have worked on them.
I really should chase down the other references for further reading.
I am happy enough to have its ideas infect .NET and Java systems, Akka and Orleans.
As I learned with Lisp, ML, Prolog, Smalltalk since the mid-80's, and safer systems languages as well, it is easier to eventually get some of the goodies, than trying to push for the whole thing.
Unfortunely that is how our industry works, until liabily becomes common, the lesser options will always win the market.
It won‘t but I don‘t get why. The thesis is genius, it even starts with an philosophy and an attempt to formalize software architecture.
Erlang was invented at Ericsson, used for certain phone switches, it was tremendously successful at what it was built to do, and still it‘s not used anymore in their newer switches.
Whatsapp (and IIRC Wechat) were built in Erlang. I think Discord is also Elixir/Rust
My take is that when the advantage to be gained is a superior architecture or set of abstractions that can be built by a small core of engineers then language choice can be an advantage. When the business needs to scale into the real world long tail of integrations, infrastructure, clients etc then throwing bodies at the problem can be an advantage and then ecosystem and talent pool outweigh original language gains.
Sharp tools are at their best in a smaller more focused team... But in software, building the right architecture can in some cases be all the difference
You're talking about two products and two companies with (initially, and still core) well defined domains.
The languages, the architectural choices, that were made by these orgs early on were allowed to "stick". There was no move fast break things, there were no pivots, there wasn't a massive cycle of "market discovery", no figuring out "product market fit".
They could have built in PHP, or Ruby, Or brain fuck and been successful I suspect. Talent and a steady hand at the helm are playing the outsized roles here.
Nit: WhatsApp was built in PHP and then people started chatting in status messages. Pivoting to be a chat system may not have been a massive pivot, but it was still a pivot.
Chat was added by running ejabberd[1], and the rest of the services (registration, status, sync) remained in PHP for some time, MMS was added as a PHP service. Services migrated to Erlang over time.
[1] Need chat? Run ejabberd is a gateway drug to running Erlang. Happened lots of places, but some then rewrite chat in another language later.
> They could have built in PHP, or Ruby, Or brain fuck and been successful I suspect.
I’ve seen enough to doubt this. The less capable a tool the higher the overhead, until you reach that critical point where you spend more time managing overhead than adding features. It’s a self reinforcing cycle which makes it exponential , and it happens more quickly than people like to admit. You have to start hiring, and hiring is always a disappointment, you grow teams, communication overhead takes over, you inevitably model the product after the organization, you start making engineering choices based on realities of the hiring market, the realization sets in that “easier to hire for” is a double edged sword as talent pools differ not just in size but also quality, your organization and tech stack ossify, and suddenly you’re in the stink.
In fact I’ve never not seen this happen.
Case In point: You flippantly added brainfuck to your list there. But obviously not: clearly you can’t build WhatsApp in brainfuck and win the market. So here we are already: there is a limit. Why, then, should ruby or PHP be on the other side of that limit?
Remember that Erlang was open-sourced as a last-ditch effort to save it, in 1998, after Ericsson internally switched to C++ and banned using Erlang on new products.
The sad fact is that the software industry as a whole relies heavily on mainstream languages, and not on educating your engineers to leverage the strengths of your specialty language. There's rationale to that, in that it costs a ton to maintain your own platform when you can just leverage a more public one. As a random example, why spend a ton to try and optimize your compiled language when you can... just use GCC (or clang nowadays). See: ongoing complaints about the performance of Go.
Erlang's silver lining is it found some measure of success outside of Ericsson, which led it to being unbanned there (and the banner to being "promoted sideways")
I truly asking, how much cost is this, have been measured?
I suspect is like the issue with fail to measure the whole thing and is just like "my guys that work in the platform also work making my apps and fixing my printer, so when they work in the platform I can`t print" kind of situation, instead of "I have a platform team (that could be like very small) and a apps team, and I ditch the printer in the bin" that I suspect is just reorg correctly the same amount of people.
> The sad fact is that the software industry as a whole relies heavily on mainstream languages, and not on educating your engineers to leverage the strengths of your specialty language.
> The sad fact is that the software industry as a whole relies heavily on mainstream languages, and not on educating your engineers to leverage the strengths of your specialty language.
Except when the language is the workhorse of a proprietary product sold to your employer. SQL is the example that escaped into the open world, but see also MATLAB and Excel formulae/VBA.
My experience is people are reluctant to bring a new vm into the fleet.
The jvm, gdb, linux perf stuff for posix apps, these are known quantities. But Erlang is different and weird. It gets some great features as a result of this, but it means most of the IT and Dev orgs are in very unfamiliar and uncomfortable territory.
As for Ericsson themselves, I don't know for certain, but my suspicion is it was driven by the chip/hardware side. They're buying network processors and other components from Broadcom, Qualcomm, ARM flavor whatever, etc, and with each new edition of these gcc will already be ready to go from the vendor, vs erlang may need to be updated to keep current. It probably got old trying to maintain what was almost a parallel OS.
because the benefits aren't that much greater, and great features in languages like erlang and haskell end up (to some extent) in popular languages (which further reduces the need to adopt such languages)
I agree, but the main feature isn’t the language, but the BEAM VM.
As far as I know there‘s no other language which focussed from the get-go on parallel, distributed deployment, has processes as first class primitives, supports hot code swapping etc.
> Wouldn't it be OK if the initial focus was something else and then there was a change of focus?
Retrofitting features into existing languages is rarely satisfying in my experience.
Sure, Python now has functional constructs, but variables are still mutable, so bizarre things can happen inside a map call.
In Erlang, immutability is baked into everything.
Message passing can be added as a library, but you don’t get the same impact as everything being a message. Does your random 3rd party library also use messages? Probably not.
Erlang is a very opinionated language and runtime with a small set of core concepts, and it is very predictable as a consequence.
Java or C++ have just about any feature you can think of, which is beneficial in some ways, but they’re also much more complex.
I haven’t worked much with BEAM/Erlang but having worked with real distributed applications I’m quite skeptical there’s much value in helping me send messages over the net between processes. That’s the trivial part of distributed systems. The hard parts of distributed computing like data consistency, consensus, or network partitions are left to the programmer. To say nothing about hardware.
Something simple like a message queue has many ways it can be implemented in a distributed system. In Erlang every actor has its own queue. Great. I highly doubt I could actually use them in a real distributed system but I’d love to see that I’m wrong.
Here's a trivial example from one of my recent jobs: We need to queue Slack messages from different producers into some kind of per-tenant outbox queue that ensures one consumer per tenant, and which consumes at a maximum rate (to stay below Slack rate limits). This queue also needs to be bounded and communicate back pressure to producers. What unique value does Erlang provide here? Usually this kind of thing is built using Go and AWS Lambdas and SQS or Kafka. Where does Erlang/BEAM help here?
Link to the excellent Jim Gray paper that Joe repeatedly cites:
https://pages.cs.wisc.edu/~remzi/Classes/739/Fall2018/Papers...
I gave an Erlang talk many years ago where I used the Tandem architecture as a comparison point, and one of the attendees happened to have worked on them.
I really should chase down the other references for further reading.
RIP, Joe. You were one of a kind.
Erlang (and quite frankly Elixir) really has always seemed like it was, in some form, ahead of its time. I really hope it spikes in popularity.
Any day now...
I am happy enough to have its ideas infect .NET and Java systems, Akka and Orleans.
As I learned with Lisp, ML, Prolog, Smalltalk since the mid-80's, and safer systems languages as well, it is easier to eventually get some of the goodies, than trying to push for the whole thing.
Unfortunely that is how our industry works, until liabily becomes common, the lesser options will always win the market.
It won‘t but I don‘t get why. The thesis is genius, it even starts with an philosophy and an attempt to formalize software architecture.
Erlang was invented at Ericsson, used for certain phone switches, it was tremendously successful at what it was built to do, and still it‘s not used anymore in their newer switches.
Why?
Whatsapp (and IIRC Wechat) were built in Erlang. I think Discord is also Elixir/Rust
My take is that when the advantage to be gained is a superior architecture or set of abstractions that can be built by a small core of engineers then language choice can be an advantage. When the business needs to scale into the real world long tail of integrations, infrastructure, clients etc then throwing bodies at the problem can be an advantage and then ecosystem and talent pool outweigh original language gains.
Sharp tools are at their best in a smaller more focused team... But in software, building the right architecture can in some cases be all the difference
You're talking about two products and two companies with (initially, and still core) well defined domains.
The languages, the architectural choices, that were made by these orgs early on were allowed to "stick". There was no move fast break things, there were no pivots, there wasn't a massive cycle of "market discovery", no figuring out "product market fit".
They could have built in PHP, or Ruby, Or brain fuck and been successful I suspect. Talent and a steady hand at the helm are playing the outsized roles here.
Nit: WhatsApp was built in PHP and then people started chatting in status messages. Pivoting to be a chat system may not have been a massive pivot, but it was still a pivot.
Chat was added by running ejabberd[1], and the rest of the services (registration, status, sync) remained in PHP for some time, MMS was added as a PHP service. Services migrated to Erlang over time.
[1] Need chat? Run ejabberd is a gateway drug to running Erlang. Happened lots of places, but some then rewrite chat in another language later.
> They could have built in PHP, or Ruby, Or brain fuck and been successful I suspect.
I’ve seen enough to doubt this. The less capable a tool the higher the overhead, until you reach that critical point where you spend more time managing overhead than adding features. It’s a self reinforcing cycle which makes it exponential , and it happens more quickly than people like to admit. You have to start hiring, and hiring is always a disappointment, you grow teams, communication overhead takes over, you inevitably model the product after the organization, you start making engineering choices based on realities of the hiring market, the realization sets in that “easier to hire for” is a double edged sword as talent pools differ not just in size but also quality, your organization and tech stack ossify, and suddenly you’re in the stink.
In fact I’ve never not seen this happen.
Case In point: You flippantly added brainfuck to your list there. But obviously not: clearly you can’t build WhatsApp in brainfuck and win the market. So here we are already: there is a limit. Why, then, should ruby or PHP be on the other side of that limit?
I doubt it.
> It won‘t but I don‘t get why.
Remember that Erlang was open-sourced as a last-ditch effort to save it, in 1998, after Ericsson internally switched to C++ and banned using Erlang on new products.
The sad fact is that the software industry as a whole relies heavily on mainstream languages, and not on educating your engineers to leverage the strengths of your specialty language. There's rationale to that, in that it costs a ton to maintain your own platform when you can just leverage a more public one. As a random example, why spend a ton to try and optimize your compiled language when you can... just use GCC (or clang nowadays). See: ongoing complaints about the performance of Go.
Erlang's silver lining is it found some measure of success outside of Ericsson, which led it to being unbanned there (and the banner to being "promoted sideways")
> it costs a ton to maintain your own platform
Truly? And considering here the option is C++?
I truly asking, how much cost is this, have been measured?
I suspect is like the issue with fail to measure the whole thing and is just like "my guys that work in the platform also work making my apps and fixing my printer, so when they work in the platform I can`t print" kind of situation, instead of "I have a platform team (that could be like very small) and a apps team, and I ditch the printer in the bin" that I suspect is just reorg correctly the same amount of people.
> The sad fact is that the software industry as a whole relies heavily on mainstream languages, and not on educating your engineers to leverage the strengths of your specialty language.
That is an excellent remark.
> The sad fact is that the software industry as a whole relies heavily on mainstream languages, and not on educating your engineers to leverage the strengths of your specialty language.
Except when the language is the workhorse of a proprietary product sold to your employer. SQL is the example that escaped into the open world, but see also MATLAB and Excel formulae/VBA.
> I don‘t get why
My experience is people are reluctant to bring a new vm into the fleet.
The jvm, gdb, linux perf stuff for posix apps, these are known quantities. But Erlang is different and weird. It gets some great features as a result of this, but it means most of the IT and Dev orgs are in very unfamiliar and uncomfortable territory.
As for Ericsson themselves, I don't know for certain, but my suspicion is it was driven by the chip/hardware side. They're buying network processors and other components from Broadcom, Qualcomm, ARM flavor whatever, etc, and with each new edition of these gcc will already be ready to go from the vendor, vs erlang may need to be updated to keep current. It probably got old trying to maintain what was almost a parallel OS.
because the benefits aren't that much greater, and great features in languages like erlang and haskell end up (to some extent) in popular languages (which further reduces the need to adopt such languages)
I agree, but the main feature isn’t the language, but the BEAM VM.
As far as I know there‘s no other language which focussed from the get-go on parallel, distributed deployment, has processes as first class primitives, supports hot code swapping etc.
Well afaik is an argument based on lack of evidence.
And why does from the get-go on parallel matter? Wouldn't it be OK if the initial focus was something else and then there was a change of focus?
And processes as first class depends what you mean: https://live.exept.de/doc/online/english/overview/basicClass...
And hot code swapping: Smalltalk.
> Wouldn't it be OK if the initial focus was something else and then there was a change of focus?
Retrofitting features into existing languages is rarely satisfying in my experience.
Sure, Python now has functional constructs, but variables are still mutable, so bizarre things can happen inside a map call.
In Erlang, immutability is baked into everything.
Message passing can be added as a library, but you don’t get the same impact as everything being a message. Does your random 3rd party library also use messages? Probably not.
Erlang is a very opinionated language and runtime with a small set of core concepts, and it is very predictable as a consequence.
Java or C++ have just about any feature you can think of, which is beneficial in some ways, but they’re also much more complex.
rarely satisfying seems vague. Maybe we need requirements that can be checked by different individuals and give repeatable results.
> immutability is baked into everything
process dictionary
I haven’t worked much with BEAM/Erlang but having worked with real distributed applications I’m quite skeptical there’s much value in helping me send messages over the net between processes. That’s the trivial part of distributed systems. The hard parts of distributed computing like data consistency, consensus, or network partitions are left to the programmer. To say nothing about hardware.
Something simple like a message queue has many ways it can be implemented in a distributed system. In Erlang every actor has its own queue. Great. I highly doubt I could actually use them in a real distributed system but I’d love to see that I’m wrong.
Here's a trivial example from one of my recent jobs: We need to queue Slack messages from different producers into some kind of per-tenant outbox queue that ensures one consumer per tenant, and which consumes at a maximum rate (to stay below Slack rate limits). This queue also needs to be bounded and communicate back pressure to producers. What unique value does Erlang provide here? Usually this kind of thing is built using Go and AWS Lambdas and SQS or Kafka. Where does Erlang/BEAM help here?
IIRC it was never used for any voice/call switching. I think it was used in one (line of) ATM switch(es).
It probably won’t, would kill too many businesses and jobs. And frankly many people don’t see its value