Journeying into the Under-dark

I was recently asked by a colleague to talk about security as it pertains to startups. The exact phrase I believe was something along the lines of “what are the first things you would do?” My simple 5 or 6 paragraph response breaking down just the top most highlights of several broad categories that all have to be considered and implemented made for a light response (I thought). Security isn’t a set of tasks, it’s a lifestyle.

Unlike in our real lives there’s a line of people standing at the door to your virtual house, and all you have to do is … nothing… and they will get in. Maybe through the mail slot, or by slipping the pins from the door jamb, or perhaps cutting through the siding and then drywall (hope you had plywood installed, not all houses do anymore), or they may even call and say there is a candy bar basket free to anyone that will simply let them see their key for a few seconds to “make sure its a safe key”… Metaphors might be a little bit much to hammer the point home so let me just start with this – if they want in they will likely get in, it might just take massive patience and time. Scary right? “But I use a firewall!” -Good for you. Sadly there are so many ways to subvert security and create an entrance; it’s an ongoing job that literally gets harder everyday. If that sounds like your thing by all means read on. If this sounds exhausting and like something you can just write a shell script to automate, you should really consider some other pursuit.

So my software is to blame and they should be the ones that fix it!? Sadly not always. There is a very small group of folks that are extremely dangerous and can exploit some of the most difficult technical hacks in the world including overflowing buffers on hardware to inject (or read) buffers to which they shouldn’t have access. The vast number of hackers have been inspired by something different though, sometimes it’s the idea of freedom of information, other times it’s based on greed, or corporate espionage, or sometimes they just want to wreak a bit of havoc. This last category doesn’t even need to have any real proficiencies other than being near a terminal that is open when you aren’t paying attention. If you watched the recent Batman movies you might have caught Michael Cain as Alfred sharing the idea “some men (people) just want to watch the world burn”. There is likely a Psychology MS at least that might help us understand what it is that makes folks hit the delete button on something that looks important when they don’t think they will get caught. At the end of the day though, you can only trust those you can actively watch. Sorry folks, no trust falls in Infrastructure; team building is awesome, but you still don’t get Root.

I always tell people you’re only paranoid if people AREN’T watching you, and in this case, they are. You may think nobody cares about your unprotected webcam watching the front door of your house, but that webcam (pick a popular vendor everyone is installing) often is a full blown computer, and sadly not capable of network updating their software. It’s essentially a low powered PC that might just get you additional access to other parts of a vulnerable network (not to mention potentially risqué pictures).

If you can’t update the device without pulling it physically down and plugging it in somewhere else… this means there are millions of these little computers with network access and a decent processor just sitting there with known exploits on them waiting to be found. Yick. Like roaches. Then they can be tacked on to a ring of botnets capable of anything from shutting down major services due to the massive number of devices that can be turned to nefarious ends or brute forcing access. The worst part is, often the hacker doesn’t even really care, they just sell it off to whomever will pay for its use.

Let me start a bit simpler. There are several top level categories you need to actively participate in (and this will change depending on your deployment type, network traffic, infrastructure size, and employee dispersion) based on the following core risks (These are buckets, not comprehensive topics. Each one could take an entire course to get a solid handle on.), which are roughly:

Direct machine based attacks

  • Buffer overflows
  • Brute force
  • Keylogger
  • Trojan
  • Virus
  • Malware

Network attacks

  • LOIC,
  • DDoS,
  • Packet Sniffing / editing
  • MitM
  • Poison network (wireless at a coffee shop)
  • Cookie Theft

Software Flaw Attacks

Such as (software flaws or intentionally malicious software):

  • Injection
  • XSS / XXE
  • Unsanitized input / Deserialization
  • Authentication encryption standard

Physical Attacks

(access.. you have to be there..) such as:

  • Rooting a machine
  • Re-installing with a keyboard shortcut on an open network
  • Pulling the drive and mining it (this is oddly very applicable to things like corporate fax machines / copiers. Always remember to pull your drives before you recycle. Ever faxed your drivers license? or Copied your tax return? Its probably still in that drive… or at least it could be)
  • Jacking into an unprotected network directly via eth

Internal attacks

  • Accidentally through getting hacked / caught in a Virus and creating an entry point to an otherwise secure network
  • Not so accidental like a disgruntled admin that builds in dead fall switches so if they don’t show up one day things go rotten

Social Engineering

Which encompasses many attacks such as:

  • Phishing through spoofed email headers and nefarious shortened redirect links that will take you to something you don’t want to give your personal info (but it came from my Dad!!!… seriously, did you think your Dad is really stuck in Aruba and needs your bank account to get out of jail? likelihood < .01%)
  • Baiting is when you get those calls saying you have won a free cruise… all you have to do is give your social security number and mothers maiden name. Still holding off? Ok, we will throw in a free big screen OLED TV for free, just refer us to three of your close relatives… yeah…. sign me up! Here is my wifes SS# too if I can just get that free golf cart you promise to ship me from Antigua.
  • Tailgating is where people slip in on the good faith of others, such as into an office building without a badge by simply asking “oh hold the door for me please.” Once inside many networks will give you raw access by just accessing a direct eth connection. But many worse things can come from this… is basically the doorway to many forms of physical access. The worst case that jumped out recently was an old friend that had someone walk in the office on a Friday when everyone was in a meeting, walked around from desk to desk and stole over 15 laptops. Ouch. What can you do with those? … all depends.
  • Evil Maid is someone who has direct and repeated access to your systems. I worked at a place several years ago that had an Evil Maid that ended up taking one of the Corporate Amexes for a spin and thieving several hundred dollars. Small shop, but was an open building… keep those digits closed!
  • Pretexting is where someone calls and creates an elaborate scenario which compels you to provide them access or trust. Such as “Joe” the IT guy calling around stating its a new requirement that all machines have to be brought up to date on the latest corporate security software pack. Just go to this URL and follow the link, or if you want … I can do it for you… just install this software and add a link for my account….. Another famous breach was performed on a security firm by an attacker that got access to the CEO’s client password via a junky homegrown CMS, contacted the internal workers claiming he was at a show and got them to drop the firewall. Whoops.

Sounds like a lot right? That’s just the tip of the iceberg. New threats emerge everyday and being the Apex predators humans are, we adapt not just to the incoming challenges (given any money to an African Prince lately?), but also to overcome those challenges with new and creative means of stealing identities, credentials, and causing harm or loss. If you enjoy a steady supply of challenges this is a great field to get into, as most people turn a little pale and clammy just thinking about keeping up with it, but somebody has to.

Ok, before I go (I know its a huge article already) I want to leave you with the top starting points for the above categories:

  • First, and this is the big one… draw it out.
    There are loads of UML modeling tools on the app store and everywhere else. Put down what your services are, how they interact with each other, which ports they use, which should be public and which users / servers should have access to them. I find putting the people / users into a spreadsheet gives me an easy way to keep track of who should be in what groups.
    If you can do this you should be able to immediately lock down a large portion of your network from undesirable access. The less folks can get to the harder it is to get a foothold (but be aware this is NOT a fix-all. Just because a service is “safe” behind a firewall doesn’t mean it’s safe. If someone breaches the wall through any of the above vectors then these un-patched machines become the next low hanging fruit. But its still the right place to start as it will keep a number of folks out to begin with.


  • Make sure your software for any OS’s as well as programs you may be running as servers are updated regularly.
    The more often they sit on old versions the higher the likelihood someone is going to find the hole.


  • Turn off any services you do not use.
    Not only will this make your machine run faster as it doesn’t have to service idling programs, it will also completely remove the ability to attack that program (with very few exceptions, but there is always one).


  • Setup a comprehensive preferably NAT based firewall for your network.
    This is in addition to turning things off, as you never know when someone is going to accidentally start a service that doesn’t need to be on. And voila you have a promiscuous email relay for example. Firewall it off so there is nothing coming in to begin with except ONLY the very specific bits you wish to allow. If you do not know what these are you should figure it out or find someone that can help you with it. Learn about NAT and reverse proxies, they will help you immensely throughout the years.


  • Turn off password based logins to anything public.
    This isn’t ideal, turn passwords off on everything if you can get away with it, but at least start with anything public. If you must have passwords make them obnoxiously unfriendly (make people beg you for an ssh key) and never tied to user accounts that are easily guessed.
    If you must have passwords its also a very very good idea to put limits on the amount of attempts that can be made before the account is locked and must perform a rest or a long wait to continue logging in. This will go a long way towards defeating brute force bots. Seriously though, learn about ssh keypairs and how basic encryption works.


  • Setup monitoring on everything that is critical to the environment – even if it doesn’t change very often.
    The fastest way to know there is a problem is to be notified. There are lots of great utilities out there to make reading logs and centralizing distributed information simpler, I’m still a big fan of Nagios for its notification capability, letting me know about core metrics, whens my SSL cert going to expire; that’s not just dangerous, but a bad experience for my users. Are disks filling up? If so maybe someone is filling them with something I don’t want there. The nice thing is it will notify you if it passes your predefined thresholds. Then if you have images setup you can pull the machine offline and spin up a clone in its place that’s clean, then go about diagnosing what’s wrong with the broken one. It also provides a nice means of scheduling “pager” duty. (I know you probably have never even seen a pager IRL but it was a thing…a horrible horrible thing). Being able to set the schedule of when, and who gets notified is super nice. You can also integrate the data into other services, and its easy to extend with all kinds of scripts in anything shell friendly.

    There are numerous other utilities to help you drill down though too. Prometheus is one of the latest on the scene allowing for really granular defined views of data over time, or if you have loads of servers / services you might check out something to help manage the amount of incoming data (so as not to DDoS yourself) with a tool like Ganglia. Whatever you choose make sure it gives you the core information about your app, network, and hosts (nodes) so you can ensure not just that they aren’t spinning out of control but when to scale as well. (What’s the point of security if no-one can access your site to begin with?)


  • Setup MFA on anything that has administrative access.
    Multi-factor authentication will make guessing regular credentials used on a daily basis exponentially more difficult. Preferably this should be performed using a hardware or virtual device such as Google Authenticator. If you choose to use a text or callback as a second factor there are numerous accounts where peoples cell phone accounts are hijacked and you then have to go fight to figure out how to get that back online, let alone getting your core services back and running! If you are using AWS this is particularly important. There are at least 2 levels of accounts, the actual “root” AWS account, then all the admins. Make SURE the root account is enabled with MFA. As a secondary note to this, use 2 devices to setup the MFA at the same time. If you only have one and it gets lost or broken you then have to fight your way back in through the red tape (this is also true of any attacker though so that’s a good thing). If you have a secondary device you sync at the same time then you at least have a fallback. This should not be a shared device.


  • Limit user permissions.
    AWS (and most cloud providers) support the idea of IAM or user based roles and privileges. If you give everyone admin privileges you will absolutely make their lives easier, but if any of their accounts is ever compromised (giving the benefit of the doubt that they at least have distinct accounts) then everything is up for grabs… the attacker now has access to your entire infrastructure. Consider what the users need access to… or what service they are setting up needs access to and how (read/readwrite/delete/etc). Isolate these accounts so they limit anything an attacker can get to, preferably to the point of uselessness. You don’t have to lock everyone from everything, but make them give you at least one solid reason they need access to whatever it is… then record it in the spreadsheet you created above.


  • Learn the difference between Hashes and Bidirectional Asymmetric keysets.
    The latter is what you want nearly always and can provide levels of encryption and security that substantially raise the bar of difficulty for intrusion. Keep in mind though in the world where all commands typed can be automated you are only ever as secure as the weakest part of your Infrastructure.  Using Secure keys and managing them on a per employee basis is the only way to go. If you pass around the main keys you are basically guaranteeing you can no longer trust that key. Not to mention you will have a much harder time identifying who was where when something happens.


  • Setup snapshots that you can recover from.
    The web providers all have these, but even if your machines are local you can use DD or other image making software to encapsulate your servers which not only lets you recover to a known good point (or start forensics on) but also gives you a nice ability to rapidly scale. Containers are the latest greatest means of doing this at a more software level but have some hiccups like everything else. One way or the other, cronjobs, AWS scheduled backups, or other… these will likely save your bacon in a number of instances at some point in the future.

Once you have cataloged and diagrammed your services, setup a secure network with locked down ports and a solid firewall, closed the ports of your unused services, ensured your software is updated everywhere, setup monitoring on all critical endpoints and service/machine metrics, setup snapshot backups and archiving, and swapped out passwords for ssh keys you can move on to the next big items.

When these top level items are covered it’s time to move on to the cookie dough you are making. These next steps include identifying and preventing software security holes from entering your code, ensuring your libraries are updated at the code level, VPN’s both for employees and for your data centers, IDS systems like Snort, creating a predefined response plan for various situations with backups to execute if someone is not available, Software exploit testing systems like OWASP Zap or Arachni, machine vulnerability scanners like Nikto2, network sniffers and much much more. All the while ignoring the guys saying they are from IT and want to help you solve your Desktop software problems (Social Engineers).

If you perform your job well you will be praised, hated, feared, reviled, and loved; sometimes in the same conversation. If it sounds like “work” you should probably think about something else, as it really only gets more intense. If the thought of constantly learning about new possible attacks, creating systems to orchestrate (and manage long term… those things add up) images and backups, learn about every kind of software language used in your stack (or considered) and their particular vulnerabilities, etc., sounds like an adventure here’s some info to get started: