Application Security & Cryptography with Scott Arciszewski

Welcome to the sixth episode of Syscast. This one is all about security, cryptography, and PHP. I am joined by Scott Arciszewski, who is a PHP developer and cryptography guy.

We discuss all things security. We talk about the OWASP top 10, SQL injection, cache timing vulnerabilities. We also talk about the CMS that he’s developing called Airship and all the security involved there.

Now, I’d like to give a special thanks to both Jeroen Flamand and HPCDude for helping with the audio quality. They removed a lot of the background noise and clicks. And you can find both of their Twitter handles in the show notes.

So thanks, guys. Now, without further delay, here’s the episode. Enjoy.

Hello there, my name is Mattias Geniar and this is the sixth episode of Syscast. Today I am joined by Scott Arciszewski. Hi Scott, how are you?

Sure. I’m Scott Arciszewski. I’ve been the Chief Development Officer at Paragon Initiative Enterprises for over a year now.

I’ve been doing PHP development and security for about 15 years. I’ve been contributing to open source for the last several, since 2013. Most people know of Paragon through some of our open source libraries, like a PHP 5 compatibility layer for the random number generator. called Random Compat.

So I came across a program called RPG Maker, which was pretty simple to use. But it… I decided I wanted to build my own website dedicated to like tutorials and articles because there was a lot of, I guess you could say politics, um, between different communities.

Um, everything wasn’t very collaborative back then. So I set up my own website and shortly, I think three weeks into it, it got hacked pretty hard. So I started over and then that kept happening and I figured out how they got in.

So I started changing on programming habits. And as time went on, the hacks stopped. But I started on the road of teaching myself how to build secure websites.

So I just kept learning more and more until a couple years ago when I discovered a problem with CodeIgniter. And I went to go tell them. And I’m like, hey, it’s a timing attack vulnerability on your Mac validation.

And they’re like… does that mean i’m like what come on everybody knows what this is it’s basic cryptography and then it turns out that uh no in fact not everybody does know what that means so it was kind of a culture shock what is obvious to you is not always that obvious to the person you’re talking to um out of interest what were the the vulnerabilities that got into your first website um oh Really basic stuff. I didn’t know how to use MySQL, so I was storing stuff inside of a file, which was in the public web root, but it had an obscure name, so I figured nobody would guess it. It was accessed by file operations, so if you knew how to escape the way the file was written, i.e. tabs and new lines, you could add new rows and create user accounts.

It didn’t use proper password hashing, so you could literally just dump in an MD5 hash and create a backdoor user. It was really terrible. This was back in 2002 when I really actually got into building websites.

So I started working professionally in the field and I was brought on board for a telecommunications company to build a safety inspection app, basically for their guys in the field to do their inspection checklists through a mobile app instead of having to do all this paperwork. And they built it in CodeIgniter, and I was looking through the code, and they had a cookie-based session storage mechanism that had an unserialized in there that was guarded by either encryption or authentication. And I found that the authentication, instead of using a proper function like PHP 5.6’s hash equals function, it just compared it with saying if this is not equal to this string, return false, otherwise, un-serialize.

For people who aren’t in the know, un-serialize is a PHP function that stores data like a class or an object as a string, and un-serialize turns that string back into an object. If you send a specially crafted string, you can control the objects that get created, and if you do it creatively, you can actually create a remote code execution vulnerability out of it. So I got really concerned because this is inside of a corporate code base, a potential remote code execution, not the most straightforward one to exploit, but nonetheless one that’s worth being concerned about.

So I patched it for them, and then I… I forget the exact order of operations that followed after that, but I ended up doing a talk at B-Sides Orlando about that and some of the vulnerabilities I found in popular PHP frameworks. And it was titled, When Frameworks Don’t.

And that was actually my real big introduction to open source.

Yeah, cryptography is kind of like a depth… It’s like a dive into madness. Things that you take for granted, like comparing strings as totally safe and something that’s just what you do.

It’s just, you know, oh, if this is equal to this, proceed to this step. You have to actually be very careful how you do it because of something called side channels. I mentioned a timing attack, which is where if you… are comparing, like, a HMAC hash of a message with a secret key, and you, you know, are comparing the message, the HMAC of the message compared to the one they sent to authenticate the message, and you are comparing that in constant time, which is a specific way of not just using equals equals.

If the first, and I think PHP goes by words, so the first four or eight bytes, depending on your architecture match, it will take slightly longer than if the first four don’t match or first eight don’t match. And in doing so, you can actually find the expected value for the HMAC hash without having to know the key. And you can actually forge messages that way. there’s a lot of ways you can introduce side channels into cryptography code.

And most developers would actually be horrified if they realize how many different ways they are. Like an if statement can be a branch-based cache timing side channel, indexing secret data like the AES function, which is a really well-known cipher. If you implement it in software, the way most people implement it, it can actually leak information through processor cache misses. you wouldn’t think to look there unless you’re a cryptographer or a security engineer.

So it’s definitely a level of complexity that most developers just don’t have the background for and would never think to, you know, oh, an attacker could exploit this. But there was a research paper back in 2014, I believe. It might have been a little bit earlier than that.

They managed to leak the AES key using OpenSSL in 65 milliseconds through cache misses.

You do what you can. If you’re dealing with secret data, like if you’re writing an encryption function, which most people probably never should because there’s just so many things you have to watch out for. make sure that your key or your ciphertext or your plaintext, you never index based off that in memory. You never say at the offset of secret key value one plus three bytes as the way something called an S-box, which is a substitution box.

Modern ciphers are based off of very simple primitives mixed together in a mathematically rigorous way. If you’re storing something in memory and you’re saying, okay, we’ll let this offset grab this value. That’s what actually causes the cache timing leak is that it’s the memory address you’re using for the operation will be stored in your L1 or L2 cache or even L3 if it’s a big operation like encrypting like a large message.

So the trick is to… I mean, there’s actually a website called cryptocoding.net, which has a list of coding rules for writing secure cryptography. And most of it’s for C developers.

If you’re doing anything with C, assembly, or any other very low-level language compared to a web application, they’re the kind of things that would be really interesting. If you’re building a web application in Python, PHP, or JavaScript… it’s probably something you trust your library to take care of for you. But that’s only for cryptography code, like Taylor’s research on flush and reload attacks and whatnot against Firefox, the web browser.

There’s really nothing you can do to defend against it. It’s one of the things that makes malware such a problem, is that if you get malware on your computer, you trust your antivirus to get rid of it, but… or you trust Windows or whatever OS you’re running to isolate it so that way you can’t spy on your other processes, but there’s only so much it can do.

So I actually don’t toe the line with most of the security industry. A lot of people say start with the OWASP 10 or the SAMS 25, start with this checklist. make sure you don’t have these particular vulnerabilities, which some things like insecure direct object reference doesn’t mean anything to an English speaker. I take a different approach.

I start with what I call a taxonomy approach, in which you look at the most general case of what causes something to be insecure, and then you just don’t do that. Case in point, like SQL injection, cross-site scripting, local and remote file inclusion, all of these… different forms of code injection vulnerabilities that have their own exploitation paths and different ways of handling them are all kind of an instance of the same core fundamental failure, and that’s to separate code from data and treat them differently. If you take data that your user provides and you run it as code, you got to form a remote code execution.

If you take code that’s in your system and return it as data, you’ve got an information leak, which is typically what breaks like ASLR and systems programming. So you always want to write your code to where you have data in one place, code in the other. To stop SQL injection, a good way to do that is what’s called a prepared statement. where you would send your SQL query to the database server, and then you would send your parameters in a separate packet.

There’s no way for the data in the packet or the parameters to corrupt your query string because it’s already in the hands of the database server. You would need to actually split the server software itself, which is probably several orders of magnitude more difficult than just inserting an apostrophe in a parameter on a web page somewhere.

Exactly, and as an attacker, the low-hanging fruit is probably the thing you go for first. And if there are so many leaky websites, they are a much easier target than you escaping your SQL parameters correctly. Definitely.

Okay. So as a developer, I think one of the things that makes this so hard is that there are so many vulnerabilities out there from client side to server side. It’s pretty much impossible to do this on your own, I think, keeping up with everything, writing secure applications, yet you try this.

You try to keep up on your own with everything. How is that working? How do you find the time to keep up with all these vulnerabilities?

I make mistakes, to be completely honest. I recently learned that SVG files, which come with the image slash SVG mime type or image slash SVG plus XML. you would think that’s an image file, but you can actually embed JavaScript directly in it. So one of the projects I was working on, it would just upload and then immediately serve an SVG file.

And if you viewed it directly, it would just run JavaScript on the user’s computer. So that’s a stored cross-site scripting vulnerability. You would never think to do that.

So one of the biggest challenges is that the standards are not necessarily your friend.

I don’t know. I’ve always, I took MIME types very literally. If it says application slash whatever, I would always format it as text slash plain and sort of write to them as, you know, via the source.

Just because certain misconfigured web servers, if you upload a PHP file, like if you have a bad Nginx configuration, it can actually run. And I’ve actually had that happen on websites when I was first learning to use Nginx. So that was one of the precautions I put in, but I would never have thought, hey, image slash SVG, you know, upload an SVG file and embed JavaScript, trick the user into viewing it, like making it look really pixelated or mosaics that way they want to view the image directly and pop, you’ve just hacked them.

So that actually came to us on HackerOne and we paid a bug bounty for that one because that was really clever and taught us something that we had no idea about.

Yeah, that one. It’s sort of a similar vulnerability. You embed two kinds of data types inside of a single file.

Mm-hmm. well that’s uh an obscure use or how do you say this in gaming uh clever use of game mechanics um not something uh you think of using uh but well yeah it works um okay so as a developer there are a lot of techniques that you should or can master um to secure your things if we look at things from the other side of the table as a sysadmin What can a sysadmin do to protect his servers or her servers from bad code from developers?

So it depends on the level of the codes written. So like a PHP application… The best advice I can give somebody who’s not actually messing with the code themselves is to keep it up to date as much as possible.

If you have Composer installed, that means CDing into their directory and typing Composer update anytime an advisory comes out or even every day, just as good practice. For system services, something that connects to a socket, probably not OpenSSH, but maybe an FTP server, or a web server if you’re running your own variant that’s not Apache and Genetics or Caddy, definitely install something like GRsec, which is a set of kernel patches which harden your Linux kernel against things like buffer overflows and heap overflows and all that other fun stuff that hackers love to use. That’s the big thing, is exploit mitigation.

I’ve only encountered a couple servers that I’ve ever… for clients or in my own experience, just helping people out with their websites that have had GRSEC installed. It’s one of those things the industry knows about, but very few people actually take the time to install.

Yeah, that’s the reality that we live in today, I think. It’s a shame, isn’t it? But at the same time, I think that points out a flaw in the way that security tools or routines or projects are being implemented.

Are they just too complex to be used by humans?

Yeah. Do you have any more tips for say protecting yourself as a sysadmin on the server level? Like how do you keep your SSH server configs?

Should you use VPN? Use sudo? What are service side things you can do?

So the two quick wins for like SSH access, like let’s say you have a team that needs to get in over SSH so they can execute commands. Don’t allow password authentication, period. Make them generate an SSH key pair and use public key authentication.

If they want to have a password on their private key, great. But a lot of the issues that have come, like the most recent timely attack against OpenSSH was only valid against password authentication, not public key. And the other one is, you know, disabled root login.

So if you have root access through sudo, you should probably be SSHing into your user account and then typing sudo dash i if you want to just run as root. I mean, that’s the lazy way. Or just typing sudo before any command that requires elevated access for better auditability and accountability.

Right. And another thing you can do that might be very helpful, which only works in certain circumstances, this is one of those usability trade-offs, is to actually use IP tables to only allow certain IP addresses to connect to your SSH port. Like if you know, for example, all your developers are going to connect to the corporate VPN before they SSH in, you kind of know that the IP address is going to be within a certain range.

They’re not going to just connect in from Beijing.

Yeah, there’s a lot of open proxies over there too. I would generally suggest a whitelist instead of blocking the country. maybe only allow a certain ISP within a certain country. Typically, whitelists are easier to maintain, but for something like outbound traffic, it becomes a usability headache for most users.

For something like SSH, it’s pretty easy to say, okay, you have to connect to the VPN first, and then you can connect to the SSH server. But for something like web traffic, people typically go with blacklists because you can’t enumerate every website your users are going to want to visit. No, indeed.

Unless you have a very controlling corporate culture, and then people are going to fight against it anyway.

Exactly. And it’s probably easier to say block Facebook and Twitter and just allow everything else, even if everything else is a really wide concept. Exactly.

One of those security through obscurity measures, which I sort of like, as long as it’s not the only defense, changing your SSH default port, how does that ring to you?

I don’t actually have strong feelings on that. I’ve heard a lot of arguments in favor of it and against it. The only real point that’s relevant to me is that if you don’t have your SSH port behind the VPN, you get less traffic of people trying to brute force it by scanning for port 22.

So you have less off logs to look through.

Indeed. So that’s one concern to keep in mind. But I still think that beats the trade-off of just getting 1,000 requests per second on your SSH port.

Even with tools like fail to ban to block those, it’s still something that your server needs to do. Whereas if you can just prevent it, well, all the better. Yeah.

Switching topics, perhaps. You created a new CMS called Airship. Could you tell me a bit about that?

How did that come to be? What’s the goal of Airship?

Yeah, so there’s actually a long internal story that led up to that. But the gist of that is we were trying to fix security concerns with WordPress and several other CMS platforms. And the correct fix for a lot of the problems we were running into was… you need to update their minimum PHP version.

And they were very staunch, digging their heels. No, we’re still going to support 5.2 because we committed to it, and they have all these reasons, like hosting providers will switch off support for newer versions of WordPress if we drop support for older versions of PHP. It’s a big mess.

And while I was banging my head in the wall going, how am I going to fix problems that upgrading PHP takes care of? How are we going to address all these concerns that could come up from that. The idea was, hey, why don’t we just build a CMS?

You can build something that’s secure, right? And I’m like, yeah, I’ve been doing that for years. So that was kind of where the idea came from.

And then as time went on and as we developed more features, I was talking to other cryptography experts. I do talk to a few personally. And one of them who I mentioned earlier, Taylor Hornby, formalized something called the Triangle of Secure Code Delivery.

And that’s a mental model for how to design a system for authentic secure automatic updates or for just code distribution in general to where neither the publisher nor the server infrastructure nor any server operators between the data that gets signed and committed basically universally, and the person who’s trying to consume it can interfere with it. They can’t inject a Trojan horse anywhere. They can’t do a silent targeted attack.

And the three properties are cryptographic signatures, so you know it came from the developer, reproducible builds, so you can take the source code and say, yes, this produces the same results, and something called user-based consistency verification, which is a fancy way of saying everybody gets the same thing.

Yeah. If you have all three properties, even if somebody managed to hack into my computer, jump the air gap into the Raspberry Pi I use for doing airship signing, and steal our private key and password that, you know, decrypts the signing key, they would have a very hard time using that to, like, let’s say they had a target of interest, like, let’s say a… Powerful Bank picked up Airship and used that for their public interface, and they wanted to hack that site specifically without announcing their presence to the rest of the network or having to compromise anybody else if you were a nation state trying to do this.

It becomes impossible to do that because in order for a new update, quote-unquote, to be accepted by the end server running Airship, it has to be something that they can verify through other peers and it has to be signed by our key, so even if you get our key and manage to push a fake update to our server, if your peer sees something different, it rejects the update and just will try again later and say, hey, is there something new? Oh, well, here’s an update, and then it goes to their peer, which you can have any number of peers, and it selects them randomly and says, do you see the same thing? And if not, it screams bloody murder and puts a bunch of stuff in your log saying, this is what we got, this is what we expected.

You know, it… It makes it very hard to say, oh, here’s an update. It’s signed by a valid key.

That’s good enough. So it’s a layer of security I like to refer to as herd immunity. In order to hack any server, you have to hack every server.

And that also allows us to safely roll out updates within an hour of their release and have it be deployed automatically. I know WordPress has a system where they have automatic security updates, but they don’t have the same properties we have. If you hack the update server, you can hack 25% of the internet.

Yeah, exactly. The system that you described sounds really cool. Is that the reason that nobody’s using it?

There has to be a downside to this, no?

Okay, first, it requires a little bit of cryptography knowledge to actually build something like this. I’ve already… laid the groundwork. We published a cryptography library that wraps a library called Libsodium last year called Halite.

One of the things it provides is a Merkle tree implementation, which is a data structure that’s very similar to Git commit hashes. It’s an append-only data structure based on cryptographic hash functions. It’s using things like Bitcoin and BitTorrent for to relate to something you might have used it in.

It makes it to where you can add to the tree, but you can’t go back and change history without having a break on the cryptographic hash function that’s used.

Yeah, it has that in a property. It just doesn’t have the proof of work because it’s not meant to be slow. Bitcoin is actually powered by brute force attacks, which is kind of silly, but when you think about it, that’s how people would try to break it.

So using it as the power source of verifying this currency and this transaction was signed by this person to this person and they had the coins to transfer them. It kind of makes sense. It’s like fighting fire with fire in that regard.

We don’t have a proof-of-work. It’s literally just a signature verification system and a decentralized way of synchronizing updates. So it’s not as resource-intensive as Bitcoin would be.

And we also use the Blake2B hash function, which is one of the SHA-3 finalists that was optimized for speed. So it’s actually a lot faster than even using SHA-256, which is what a lot of them use. And I think Bitcoin is one of them.

That’s cool. Why do you think systems like WordPress, they have these over-the-wire or over-the-air updates. They don’t use this kind of cryptography or validation.

Why do you think that is? Is it merely the lack of crypto knowledge internally?

I think that’s a big part of it. I’ve reached out to some of the experts in PHP security, which… A lot of people think that doesn’t exist, but there are security experts in the PHP community.

And I haven’t really found anybody who has the kind of knowledge to duplicate the kind of work I’ve done. Because I do look in the community for peers and people I can talk to that can give me feedback on my ideas in case something’s completely bonkers. But so far, like…

WordPress, they have people who are really knowledgeable on, like, file systems and all these other more normal web application vulnerabilities. But if you ask them to explain, like, oh, I don’t know, a padding Oracle attack on RSA, they’re going to give you, like, a blank stare and go, I don’t know what that means.

Oh, for sure. There are plenty. Yeah, you can actually have a very successful career of just researching those.

And you’ll never have a dull moment because there’s always new ways to attack systems and there’s new ways systems are vulnerable without ever having to touch crypto.

So that is one of the elements of why they don’t have this. Nobody’s really written it before, so they can’t just go, oh, I’m going to go grab this and use it in our implementation. But another problem is…

Quite simply, it’s a lot of work. I spent, with the rest of the Paragon team, I’d say probably a year and a half of development hours just in getting Airship ready to go.

Well… I would say about half of that was spent on the cryptography and the verification because I spent a lot of time making sure that it’s not vulnerable to any kind of attacks. Like I said, I’m using Live Sodium to handle the low-level crypto primitives.

So I had to make processes for getting that to install easily. Recently, we published an AMI on the AWS Marketplace. I had to actually write build scripts so I could take a… blank debbing an image and get LiveStudio installed and get the PHP extension installed.

There’s a lot of work.

And that’s a big part of why we decided to make Airship a free software project. It’s that way people who, they can’t get their head above water when it comes to security problems can literally say, oh, hey, here’s a CMS project we can start with that takes care of all these concerns. They use prepared statements without PHP’s default, which is emulated prepared statements.

It uses state-of-the-art cryptography. It uses the argon2i hash function. It doesn’t use MD5 code. iterated 8,000 times like WordPress does.

It actually makes it very hard for somebody who manages to dump the passwords from the database to actually start cracking them. And they don’t even have to write a line of that security code. It’s just built in.

So at Paragon, we really do want to make the internet a more secure place. Several of us have actually been the victims of data breaches, like banks getting hacked, universities getting hacked, and our social security numbers in the hands of unknown attackers. And part of the problem is that software just isn’t secure.

You can try to solve this by offering credit monitoring or adding linking boxes that offer firewall features and intrusion detection systems, but ultimately that’s all reactive and Firewalls are blacklist-based, so if you don’t know about it, you can’t stop it. There’s a lot of blind spots in security, and we believe that by making software more secure, we’ll stop a lot of these events from ever happening in the first place. And what better place to start than PHP, which is run on, by some estimates, 80% of the internet.

Yeah, indeed, which you mentioned a while back too. PHP may not have the best reputation from the outside of being a secure developing language. What’s your take on this?

Is that correct? Is PHP insecure by nature?

I think the biggest security problem with PHP is other developers. Like I said, a lot of people just can’t get their head above water with security. You’re worrying about making sure you always escape your inputs before you concatenate your user data with your SQL query, even though there’s a better way.

But you might not know about that because you’re too busy trying to put out fires. So historically, there’s been a huge lack of knowledge transfer between security experts and PHP developers. One of the things that we tried to do last year was actually go on the websites like Stack Overflow, find a bunch of answers, especially with cryptography, that had really bad advice, and actually either edit them to be better or just offer another answer if we could.

And that ended up cleaning up the top Google results for PHP encryption and other related topics like PHP, random string. So now if you go on there and type that into Google, your first results you get aren’t some random thing that says… oh, use this weak random number generator, it’s fine, without actually saying that.

Um, I’ve actually contacted companies individually that were running like versions of PHP BB from like 2006. And this was like last year that had known cross-site scripting and SQL injection vulnerabilities. Like I think one of them was running a version of e-bulletin, which had a SQL injection in the search field, like something that’s really easy to access even without a user account.

And they never bothered to patch it. And this was a human resources company.

Yeah, true. I think in part what gives PHP sometimes this bad reputation is that it’s such an easy language to get started in. It was my first go-to language.

I dare not look at my first code that I ever wrote. It also got hacked twice. with SQL injection so it’s so easy to make something that is that is insecure but that’s more an implementation problem than say a problem with PHP the language right so yeah if you’re if you’re a developer or a good developer and you want to get better at learning about security or cryptography what should be the first steps what should I learn if I want to get better at this

So we actually have a couple of blog posts on the Paragon Initiative Enterprises website that I typically point people to as the very first step. And that’s things like learning definitions, like the difference between encryption and authentication, what a password hash is versus a regular hash. Once you understand basic concepts, if you especially want to learn cryptography, I would jump headfirst into the CryptoPalace.com challenges. that was put on by Modisano before they became part of NCC Group.

They’re a really great hands-on way to learn how cryptography works and how to break protocols that use cryptography. You can use AES insecurely. AES isn’t insecure, but your protocol is.

And you can actually make it to where if you type in the letter A a bunch of times and then put in a specific character, you can study how it affects the output. And then when you change your output, you can make that go back and produce an input. When it gets decrypted, that is what you want it to be.

And most people, when they deal with encryption, they don’t think about things like, what if they flip a bit in the middle of the message? How’s that going to look on the other end? They just assume, oh, it’s going to be garbage.

It’s not going to work. But, I mean, there’s an encrypted messaging app called Telegram that late last year came out that it was vulnerable to these kinds of chosen ciphertext attacks where you change the way the encrypted message is and you can actually use this to slowly decrypt the original message without having access to either key.

That’s a good question. One I’m afraid I don’t really have the answer to. I try to help people where I can, but if you’re building like a… just a random application, you might not think, hey, I’m encrypting data.

I need to make sure that I encrypt and then authenticate the ciphertext. It’s very easy to not know about that kind of concern.

If experts are making mistakes like this, how do you expect the average developer to succeed? Telegram’s kind of a special case. their attitude towards security researchers who tell them, hey, this is a bad design, you might want to revisit that. This is either silence or arrogance.

They posted on Hacker News in response to a critic, oh, well, our protocol was designed by six PhDs and ACM champions. And his response was, okay, well, you’re not encrypting, not authenticating. There are rules of thumb in cryptography, and one of them is called the cryptographic doom principle. wherein if you receive an encrypted message and the first thing you do isn’t verify the integrity of the message that hasn’t been tampered with over the wire, you’re going to create, inevitably, some kind of problem down the line that’s going to either completely or partially break the security of your protocol.

And they still use the same protocol that’s been criticized years ago. They just fixed some of the specific bugs that were found in that research paper I mentioned last year.

Yeah. Culture is a big part of security. We can say it’s about code.

We can say it’s about all these other processes you have to have in place, but fundamentally it comes down to how do you treat the information about a design flaw that you might have written and you might believe is good enough, but somebody comes along who might be a complete stranger and says, hey, this is completely broken and here’s why. you know, how do you treat these kind of concerns? Do you fix the problem and thank them? Do you argue with them about whether or not it’s a valid concern and get into these long flame war type conversations over, oh, well, this isn’t a practical vulnerability, so we’re not going to worry about this, but you spend all that energy arguing.

I tend to go on the more just fix it side of things, even if I think it’s kind of lame, like here’s a specific vulnerability that only affects Internet Explorer users when most of the people that… that use your application are Chrome users and Firefox users, it’s kind of like, okay, well, that’s Microsoft’s problem. They’re the one who created that specific misbehavior. But I’m still going to fix the problem.

Yeah, it’s being able to put aside your ego and looking at the bigger picture or your project or whichever it is you’re working on and looking at it from the security point of view, how to make that more secure. But at the same time, I think it’s probably not that easy. There are two ways to give someone feedback or critique.

There is the online way of doing things, which usually involves a lot of name-calling and swear words. Or there’s the good kind. I think what you did recently with some Agento vulnerabilities that you found, it’s really outlining the problem, saying what’s bad about it and what you should do.

That’s constructive criticism. That’s something that you can actually do something with as a developer.

Of course, give them time to fix it. But yeah, I think that’s the difference between approaching things. If you were to go all YouTube anonymous comments style with just a lot of exclamation marks and all caps, it’s very easy to, well, in Dutch we have a saying to bump someone against the chest.

I don’t know if that’s the same in English. But you get egos, they match, they fight. In the end, you have people fighting, no longer technical discussions.

And those aren’t very constructive. I typically put that all in the category of I’m more, and I just avoid them if I can, unless I’m the person who gets dragged into it directly. And then I try to defuse situations and say, here’s the technical issues that matter.

Like in the PHP community, there’s a lot of bickering back and forth right now about like codes of conduct and the PHP framework interoperability group. And it’s just a big mess. And I’ve basically, my way of dealing with that is literally not giving it any time or attention other than like the past 30 seconds where I mentioned it.

The discussions don’t involve technical matters. They involve human matters. I have no input as an outsider.

But a lot of people, when they see these discussions, they just jump on them and they’re like, oh, we need to sink our teeth into it. This is a matter that really matters to us because ultimately people are behind technology.

Yeah, I agree. It’s also those kind of discussions that probably give, well, in this case, PHP, but it could have been Ruby or Node, whatever. It gives the community a bad name.

If they only see fighting from the outside, it stalls developments. It reeks of flame wars. It’s a shame.

Yeah, and very little press is going to be given to some of the things PHP does, right? Recently, I started an effort to deprecate a library called Imcrypt, which has been abandoned for about a decade now. And basically for PHP to stop supporting it in a future version and to market as deprecated.

And that actually passed. That’s cool. Yeah.

So we’re going to be moving towards better security for everybody. And that got less headlines than some of the issues between different conflicting personalities and personality types. And that’s a shame because here’s a bunch of great improvements to PHP.

7.0 gave us a random number generator that works regardless of your operating system and just does everything the correct way out of the box. You don’t have to worry about is it strong or not. You just literally invoke it.

And if it doesn’t work, it throws an exception, which you can catch or you can just let it crash your application until you fix your environment.

It’s a shame that those things don’t get the attention that they deserve. PHP 7 is all about more speed, less CPU cycles, less memory, and the security parts. If they make it to the top 10 list, it’s an achievement.

The removal of Encrypt, any idea which version they are targeting for that?

So the deprecation is in 7.1, which comes out later this year. And the plan is for 7.2 or 8.0, whichever the next version is, I’m pretty sure it’s going to be 7.2. It’s actually just going to remove it from the core and put it into the extension repository.

So you can still install it through PECL. I think they pronounced that Peckle, but I’ve heard mixed feedback from people when they say it like that.

Okay, so you’ll still be able to install it through Peckle, but it won’t be part of the official core anymore, which is a good thing because that means that most people who don’t install Peckle extensions won’t be able to use it. And then we can focus on moving existing codebases like Laravel switch to OpenSSL. Several other frameworks that have encryption are doing the same.

And I think that’s a good move because Encrypt is a lot slower than OpenSSL. And it relates back to the earlier thing I was mentioning about cache timing vulnerabilities. Modern processors actually have hardware gates that provide AES encryption.

And by doing that, you completely sidestep the cache timing vulnerabilities. You can get that through OpenSSL. You can’t get that through Encrypt.

So just small things like that add layers of security that developers might not even know about. And if they do know about it, they might not appreciate the reduction in risk they just got. But it’s definitely a measurable change.

It’s true. Then there’s the other side of the coin. There are the WordPresses of the world, which I think is a fantastic CMS, and they deserve a lot of praise.

But I think you mentioned it earlier as well. They still support PHP 5.2. It’s things like that that prevents innovation if you keep on…

As a CMS supporting such ancient versions, it becomes really hard to cut the cord and say, okay, from now on, it’s PHP 7 only. We go all in for security. It’s a tricky balance, I think, to maintain.

Yeah, there is definitely a balance between what is called neomania, which is always going for the newest thing all the time, and stagnation, which in technology, things are always going to lean towards neomania as a center of balance just because technology is always progressive. sitting in the same version of PHP or the same version of Linux that you’ve been running for 10 years might make sense to some people. But from a technology perspective, that’s one of those what-are-you-doing-stop kind of situations. That’s why long-term support of Linux is only five years.

For Ubuntu, anyway. But for PHP, you get three years. 5.6 is kind of weird because they voted to extend it, but you have two years of mainstream support and then a year of security patches and then you’re out of luck.

If there’s a vulnerability, you’re not getting a fix. You need to update your… your code base. And it’s better to do that proactively than when you’re under the gun, where it’s like, well, PHP 5.5 has a remote code execution vulnerability.

It can take over your server. You should update to 5.6. And then they say, but our code doesn’t work on 5.6.

We haven’t made it compatible yet. Then you’re in a between a rock and a hard place because you weren’t proactive enough.

Right. So that’s another one of the things we wanted to do with Airship was actually to establish a new norm where… Airship 1 will support PHP 7.0 until it reaches end of life.

And then a little bit longer because we didn’t get it out the gate right at 7.0’s release. But we have the same three-year release cycle that PHP does. And when support’s over, it’s over.

You either upgrade your code or, you know… Suffer the consequences. Exactly.

We don’t, we aren’t going to backport fixes publicly because we want to keep people on the latest versions so that way they can keep themselves secure. So we don’t have to ever find anybody who uses our software between that rock and a hard place. It’s not a good place to be.

And we’ve had a couple of people who’ve come to us for help that have been in those situations. And it’s like, we can help you migrate your code, but if you don’t want to put the time into that, there’s very little we can do to help you. You know, if a 5.2 vulnerability comes out, a lot of WordPress sites that are still on 5.2 they’re probably going to get toast.

And some of them will never know they’re compromised. They’ll just become part of a botnet and then silently do whatever evil things the person controlling it wants them to do.

Indeed. Yeah, we’ll have to see how that plays out. At this point, I think the PHP code itself is still a lot more attractive to attackers than, say, the PHP core, the language.

And as long as that remains, I’m not going to say everyone is safe on older PHP versions, but it could have been a lot worse.

Yeah, the low-hanging fruit is definitely the code written in PHP, not the interpreter. interpreter bugs do happen. Almost every patch release that I’ve seen in the past year has a CVE attached somewhere. But the silver lining is that most of those, you have to have specific PHP code to exploit it.

Like if you’re using Unserialize and you’re living dangerously in the first place, somebody could break out of the PHP interpreter and run their shellcode as just a system process. Or as an application process, excuse me. Instead of just PHP code being interpreted, which is already pretty bad.

You can do some pretty evil things just from PHP code execution.

Oh, and in very surprising ways as well. Just having a couple of regex possibilities can give you some kind of execution. It’s really surprising how many ways PHP offers to pop a shell, which is a good thing for the language.

It offers a lot of, how do you say, creative possibilities, but the same goes for attackers.

I mentioned a GitHub repository of a couple of hacked sites that we found where people have uploaded simple PHP shells or interesting code. It really isn’t that complicated. What we see mostly is abuse of really common vulnerabilities because those are still, as you mentioned, the low-hanging fruit.

That is what gets you hacked nowadays.

Oh yeah, in the Airship documentation, I included an example Apache configuration, Nginx configuration, and caddy file. And I think the caddy file is only like six lines long, and the other ones are like 40 or 50. And it gives you the same security, it gives you the same URL rewriting, it passes it to PHP, Fast Process Manager.

It’s, for all intents and purposes, the exact same configuration. It’s just in caddy, it’s like literally six lines. And most of the lines are like a character, like a brace.

Yep, that is the way software should be designed, in my opinion. It should be secure by default. It shouldn’t bind people’s hands.

One of the things you can do with Airship is you can actually turn off security updates, or you can say… I don’t want to trust Paragon Initiatives’ key. I want to trust my own, and I want to be responsible for pushing updates to my servers.

You can do that by changing the configuration file. But out of the box, you get all the security that we offer, and it’s easier that way.

About your tip, the caddy web server, Matt Holt, the person that created or started that one, was the very first guest of this show. I really appreciated that episode. If you’re listening, or perhaps you, Scott, if you’re interested, the episode is available online where he discusses how it got started, how we… started in open source, what the ideas and the plans are.

I think it was right before the big 0.9 release where they pretty much threw away everything of Caddy and rebuilt it from scratch with modules in mind. I think it was a really big rewrite.

Yeah, I saw the code when I was looking at it. The AWS images actually ship with Cathy 0.9, and when I was testing it originally, I was using 0.8.3. So I was looking at it to see if they did anything really weird or funky between the two, and it found nothing wrong with it.

Otherwise, I reported it to them.

The absolute best way to get a hold of me is through Twitter. My employer one or my own personal one, I monitor both of them. Paragons is Paragon IE.

My Twitter account personally is, it’s Cypher Coder, but instead of PH, it’s PHP. because I mostly deal with PHP security in my day-to-day.

You’re very welcome to the listeners. If you like this show, feel free to share it on your social media, leave a rating on iTunes or wherever you get your podcasts. It helps grow the show and it shows appreciation, which I definitely like.

So thank you, everyone. Take care and I’ll talk to you next time. Bye bye.