Security Advice to IoT Firmware Engineers
Last Friday (October 16, 2016), a major DDoS attack brought down a number of sites across the Internet. My own employer was amongst those affected by the wide spread DNS outage.
It turns out that the sheer scale (millions of unique botnet members) was made possible by the IoT, and rather shoddy engineering practices.
Its time for device manufacturers and firmware engineers to "grow up", and learn how to properly engineer these things for the hostile Internet, so that they don't have to subsequently issue recalls when their customers' devices are weaponized by attackers without their owners knowledge.
This blog is meant to offer some advice to firmware engineers and manufacturers in the hope that it may help them prevent their devices from being used in these kinds of attacks in the future.
The idea of using a simple default password and user name, like "admin/admin", is a practice from the 90's, and is intended to facilitate service personnel, and eliminate management considerations from dealing with many different passwords. Unfortunately, this is probably the single biggest problem -- bad usernames and passwords. Its far worse in an IoT world, where there are many thousands, or even millions, of devices that have the same user name and password.
The proper solution is to allocate a unique password to each and every device. Much like we already do manage unique MAC addresses, we need every device to have a unique password. (Critically, the password must not be derived from the MAC address though.)
My advice is to simply have a small amount of ROM that is factory burned with either a unique password, or a numeric key that can be used to create one. (If you have enough memory to store a dictionary in generic firmware -- say 32k words, you can get very nice human manageable default passwords by storing just four 16-bit numbers, each representing an index into the dictionary (so only 15 bits of unique data, but thats 60 bits of total entropy, which is plenty to ensure that every device has its own password -- and only requires storing a 64-bit random number in ROM.)
Then you have nice human parseable passwords like "bigger-stampede-plasma-pandering". These can be printed on the same sticker that MAC passwords are typically given. (You could also accept a hexadecimal representation of the underlying 64-bit value, or just use that instead of human readable passwords if you are unable to accommodate an English dictionary. Devices localized for use in other countries could use locale-appropriate dictionaries as well.)
Second, IoT devices should inject a minimum delay after password authentication attempts (regardless of whether successful or otherwise). Just a few seconds is enough to substantially slow down dictionary attacks against poorly chosen end-user passwords. (2 seconds means that only 1800 unique attempts can be performed per hour under automation - 5 seconds reduces that to 720. It will be difficult to iterate a million passwords against a device that does this.)
User chosen passwords should not be a single dictionary word; indeed, the default should be to use a randomly generated password using the same dictionary approach above (generate a 64-bit random number, break into chunks, and index into a stock dictionary). It may be necessary to provide an end-user override, but it should be somewhat difficult to get at by default, and when activate should display large warnings about the compromise to security that user-chosen passwords typically represent.
IoT devices generally know the network they are on; if the device has a separate management port or LAN-only port (like a WiFi Router), it should only by default allow administrator access from that port.
Devices with only a single port, or that exist on a WiFi network, should prevent administrator access from "routed" networks, by default. That is, devices should not allow login attempts from a remote IP address that is not on a local subnet, by default. While this won't stop many attacks (especially those on public WiFis), it makes attacking them from a global botnet, or managing them as part of a global botnet, that much harder. (Again, there has to be a provision to disable this limitation, but it should present a warning.)
Use of unsecured channels (HTTP or telnet) is unacceptable in this day and age. TLS and/or SSH are the preferred ways to do this, and will let your customers deploy these devices somewhat more securely.
Devices should disable any network services that are not specifically part of the service they offer, or intrinsic to their management. System administrators have known to do this on systems for decades now, but it seems some firmwares still have stock services enabled that can be used as attack vectors.
This one is probably the hardest. mDNS and device discovery over "standard" networks is one of the ways that attackers find devices to target. Its far far better to have this disabled by default -- if discovery is needed during device configuration, then it can be enabled briefly, when the device is being configured. Having a "pairing" button to give end-users the ability to enable this briefly is useful -- but mDNS should be used only with caution.
Devices often want to call-home for reporting, or web-centric command & control. (E.g. remote management of your thermostat.) This is one of the major attack vectors. (If you can avoid calling home altogether, this is even better!)
Users must be able to disable this function (it should be disabled by default in fact). Furthermore, the channels must be properly secured entirely through your network, with provision for dealing with a compromise (e.g. leaked private keys at the server side). Get a security expert to review your protocols, and your internal security practices.
Building local mesh networks of devices, e.g. to create a local cloud, means having strong pairing technology. The strongest forms of this require administrator action to approve -- just like pairing a bluetooth keyboard or other peripheral.
If you want to automate secure mesh provisioning, you have to have secure networking in place -- technologies like VPN or ZeroTier can help build networking layers that are secure by default.
The roadside is littered with the corpses of protocols and products that attempted to invent their own protocols or use cryptography in non-standard ways. The best example of this is WEP, which took a relatively secure crypto layer (RC4 was not broken at the time), but deployed it naively and brokenly. RC4 got a very bad rap for this, but it was actually WEP that was broken. (Since then, RC4 itself has been shown to have some weaknesses, but this is relatively new compared to the brokenness that was WEP.)
Its an old adage that "security by obscurity is no security at all". Yet we often see naive engineers trying to harden systems by making them more obscure. This really doesn't help anything long term, and can actually hinder security efforts by giving a false sense of security or creating barriers to security analysis.
Get an independent security expert to audit your work. Special focus should be paid to the items pointed out above. This should include a review of the product, as well as your internal practices around engineering, including secure coding, use of mitigation technologies, and business practices for dealing with keying material, code signing, and other sensitive data.
It turns out that the sheer scale (millions of unique botnet members) was made possible by the IoT, and rather shoddy engineering practices.
Its time for device manufacturers and firmware engineers to "grow up", and learn how to properly engineer these things for the hostile Internet, so that they don't have to subsequently issue recalls when their customers' devices are weaponized by attackers without their owners knowledge.
This blog is meant to offer some advice to firmware engineers and manufacturers in the hope that it may help them prevent their devices from being used in these kinds of attacks in the future.
Passwords
Passwords are the root of most of the problems, and so much of the advice here is about improving the way these are handled.
No Default Passwords
The idea of using a simple default password and user name, like "admin/admin", is a practice from the 90's, and is intended to facilitate service personnel, and eliminate management considerations from dealing with many different passwords. Unfortunately, this is probably the single biggest problem -- bad usernames and passwords. Its far worse in an IoT world, where there are many thousands, or even millions, of devices that have the same user name and password.
The proper solution is to allocate a unique password to each and every device. Much like we already do manage unique MAC addresses, we need every device to have a unique password. (Critically, the password must not be derived from the MAC address though.)
My advice is to simply have a small amount of ROM that is factory burned with either a unique password, or a numeric key that can be used to create one. (If you have enough memory to store a dictionary in generic firmware -- say 32k words, you can get very nice human manageable default passwords by storing just four 16-bit numbers, each representing an index into the dictionary (so only 15 bits of unique data, but thats 60 bits of total entropy, which is plenty to ensure that every device has its own password -- and only requires storing a 64-bit random number in ROM.)
Then you have nice human parseable passwords like "bigger-stampede-plasma-pandering". These can be printed on the same sticker that MAC passwords are typically given. (You could also accept a hexadecimal representation of the underlying 64-bit value, or just use that instead of human readable passwords if you are unable to accommodate an English dictionary. Devices localized for use in other countries could use locale-appropriate dictionaries as well.)
Mandatory Authorization Delay
Second, IoT devices should inject a minimum delay after password authentication attempts (regardless of whether successful or otherwise). Just a few seconds is enough to substantially slow down dictionary attacks against poorly chosen end-user passwords. (2 seconds means that only 1800 unique attempts can be performed per hour under automation - 5 seconds reduces that to 720. It will be difficult to iterate a million passwords against a device that does this.)
Strong Password Enforcement
User chosen passwords should not be a single dictionary word; indeed, the default should be to use a randomly generated password using the same dictionary approach above (generate a 64-bit random number, break into chunks, and index into a stock dictionary). It may be necessary to provide an end-user override, but it should be somewhat difficult to get at by default, and when activate should display large warnings about the compromise to security that user-chosen passwords typically represent.
Networks
Dealing with the network, and securing the use of the network, is the other part of the problem that IoT vendors need to get right.
Local Network Authentication Only
IoT devices generally know the network they are on; if the device has a separate management port or LAN-only port (like a WiFi Router), it should only by default allow administrator access from that port.
Devices with only a single port, or that exist on a WiFi network, should prevent administrator access from "routed" networks, by default. That is, devices should not allow login attempts from a remote IP address that is not on a local subnet, by default. While this won't stop many attacks (especially those on public WiFis), it makes attacking them from a global botnet, or managing them as part of a global botnet, that much harder. (Again, there has to be a provision to disable this limitation, but it should present a warning.)
Encrypted Access Only
Use of unsecured channels (HTTP or telnet) is unacceptable in this day and age. TLS and/or SSH are the preferred ways to do this, and will let your customers deploy these devices somewhat more securely.
Secure All Other Ports
Devices should disable any network services that are not specifically part of the service they offer, or intrinsic to their management. System administrators have known to do this on systems for decades now, but it seems some firmwares still have stock services enabled that can be used as attack vectors.
Don't Advertise Yourself
This one is probably the hardest. mDNS and device discovery over "standard" networks is one of the ways that attackers find devices to target. Its far far better to have this disabled by default -- if discovery is needed during device configuration, then it can be enabled briefly, when the device is being configured. Having a "pairing" button to give end-users the ability to enable this briefly is useful -- but mDNS should be used only with caution.
Secure Your Channel Home
Devices often want to call-home for reporting, or web-centric command & control. (E.g. remote management of your thermostat.) This is one of the major attack vectors. (If you can avoid calling home altogether, this is even better!)
Users must be able to disable this function (it should be disabled by default in fact). Furthermore, the channels must be properly secured entirely through your network, with provision for dealing with a compromise (e.g. leaked private keys at the server side). Get a security expert to review your protocols, and your internal security practices.
Mesh Securely
Building local mesh networks of devices, e.g. to create a local cloud, means having strong pairing technology. The strongest forms of this require administrator action to approve -- just like pairing a bluetooth keyboard or other peripheral.
If you want to automate secure mesh provisioning, you have to have secure networking in place -- technologies like VPN or ZeroTier can help build networking layers that are secure by default.
Don't Invent Your Own Protocols
The roadside is littered with the corpses of protocols and products that attempted to invent their own protocols or use cryptography in non-standard ways. The best example of this is WEP, which took a relatively secure crypto layer (RC4 was not broken at the time), but deployed it naively and brokenly. RC4 got a very bad rap for this, but it was actually WEP that was broken. (Since then, RC4 itself has been shown to have some weaknesses, but this is relatively new compared to the brokenness that was WEP.)
General Wisdoms
Next we have some advice that most people should already be aware of, but yet bears repeating.
Don't Rely on Obscurity
Its an old adage that "security by obscurity is no security at all". Yet we often see naive engineers trying to harden systems by making them more obscure. This really doesn't help anything long term, and can actually hinder security efforts by giving a false sense of security or creating barriers to security analysis.
Audit
Get an independent security expert to audit your work. Special focus should be paid to the items pointed out above. This should include a review of the product, as well as your internal practices around engineering, including secure coding, use of mitigation technologies, and business practices for dealing with keying material, code signing, and other sensitive data.
Comments